2023-02-12 03:58:30,497 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5 2023-02-12 03:58:30,508 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClientSimple timeout: 13 mins 2023-02-12 03:58:30,521 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-12 03:58:30,522 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/cluster_9ba13d3e-ed14-b1d2-0f74-8c456c2a06ab, deleteOnExit=true 2023-02-12 03:58:30,522 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-12 03:58:30,523 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/test.cache.data in system properties and HBase conf 2023-02-12 03:58:30,524 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/hadoop.tmp.dir in system properties and HBase conf 2023-02-12 03:58:30,524 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/hadoop.log.dir in system properties and HBase conf 2023-02-12 03:58:30,525 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-12 03:58:30,525 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-12 03:58:30,526 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-12 03:58:30,623 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-12 03:58:30,953 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-12 03:58:30,957 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-12 03:58:30,957 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-12 03:58:30,957 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-12 03:58:30,958 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-12 03:58:30,958 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-12 03:58:30,958 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-12 03:58:30,958 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-12 03:58:30,959 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-12 03:58:30,959 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-12 03:58:30,960 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/nfs.dump.dir in system properties and HBase conf 2023-02-12 03:58:30,960 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/java.io.tmpdir in system properties and HBase conf 2023-02-12 03:58:30,960 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-12 03:58:30,961 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-12 03:58:30,961 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-12 03:58:31,374 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-12 03:58:31,377 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-12 03:58:31,593 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-12 03:58:31,744 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-12 03:58:31,759 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-12 03:58:31,789 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-12 03:58:31,814 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/java.io.tmpdir/Jetty_localhost_localdomain_39533_hdfs____zaoumh/webapp 2023-02-12 03:58:31,949 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:39533 2023-02-12 03:58:31,983 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-12 03:58:31,984 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-12 03:58:32,318 WARN [Listener at localhost.localdomain/46179] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-12 03:58:32,416 WARN [Listener at localhost.localdomain/46179] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-12 03:58:32,442 WARN [Listener at localhost.localdomain/46179] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-12 03:58:32,451 INFO [Listener at localhost.localdomain/46179] log.Slf4jLog(67): jetty-6.1.26 2023-02-12 03:58:32,456 INFO [Listener at localhost.localdomain/46179] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/java.io.tmpdir/Jetty_localhost_39339_datanode____.85p0iu/webapp 2023-02-12 03:58:32,536 INFO [Listener at localhost.localdomain/46179] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39339 2023-02-12 03:58:32,792 WARN [Listener at localhost.localdomain/44387] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-12 03:58:32,800 WARN [Listener at localhost.localdomain/44387] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-12 03:58:32,805 WARN [Listener at localhost.localdomain/44387] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-12 03:58:32,807 INFO [Listener at localhost.localdomain/44387] log.Slf4jLog(67): jetty-6.1.26 2023-02-12 03:58:32,814 INFO [Listener at localhost.localdomain/44387] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/java.io.tmpdir/Jetty_localhost_40157_datanode____xbx6ny/webapp 2023-02-12 03:58:32,907 INFO [Listener at localhost.localdomain/44387] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40157 2023-02-12 03:58:32,935 WARN [Listener at localhost.localdomain/35935] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-12 03:58:32,972 WARN [Listener at localhost.localdomain/35935] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-12 03:58:32,975 WARN [Listener at localhost.localdomain/35935] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-12 03:58:32,977 INFO [Listener at localhost.localdomain/35935] log.Slf4jLog(67): jetty-6.1.26 2023-02-12 03:58:32,989 INFO [Listener at localhost.localdomain/35935] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/java.io.tmpdir/Jetty_localhost_39485_datanode____.5mdqxa/webapp 2023-02-12 03:58:33,102 INFO [Listener at localhost.localdomain/35935] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39485 2023-02-12 03:58:33,116 WARN [Listener at localhost.localdomain/46483] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-12 03:58:33,256 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf718265a82d304a4: Processing first storage report for DS-c7dfbda7-cd8b-47ec-a0b0-9d3e9f0ea3b0 from datanode e4e7b76d-7663-47f8-a739-560eaae89347 2023-02-12 03:58:33,257 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf718265a82d304a4: from storage DS-c7dfbda7-cd8b-47ec-a0b0-9d3e9f0ea3b0 node DatanodeRegistration(127.0.0.1:38689, datanodeUuid=e4e7b76d-7663-47f8-a739-560eaae89347, infoPort=46777, infoSecurePort=0, ipcPort=46483, storageInfo=lv=-57;cid=testClusterID;nsid=416442865;c=1676174311436), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-02-12 03:58:33,258 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x33b88b2285fc580e: Processing first storage report for DS-3164a8a2-6234-417e-9b99-5b810f05d9c5 from datanode f058bc0e-c917-4e0e-b523-a6533f21d9b8 2023-02-12 03:58:33,258 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x33b88b2285fc580e: from storage DS-3164a8a2-6234-417e-9b99-5b810f05d9c5 node DatanodeRegistration(127.0.0.1:43011, datanodeUuid=f058bc0e-c917-4e0e-b523-a6533f21d9b8, infoPort=38697, infoSecurePort=0, ipcPort=35935, storageInfo=lv=-57;cid=testClusterID;nsid=416442865;c=1676174311436), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-12 03:58:33,258 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xa2578157796d80df: Processing first storage report for DS-293d7a97-f507-4bd5-aab1-538ffe568afe from datanode d006747b-34dc-492d-8dd2-01dc9639e0d9 2023-02-12 03:58:33,258 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xa2578157796d80df: from storage DS-293d7a97-f507-4bd5-aab1-538ffe568afe node DatanodeRegistration(127.0.0.1:37239, datanodeUuid=d006747b-34dc-492d-8dd2-01dc9639e0d9, infoPort=44869, infoSecurePort=0, ipcPort=44387, storageInfo=lv=-57;cid=testClusterID;nsid=416442865;c=1676174311436), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-12 03:58:33,258 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf718265a82d304a4: Processing first storage report for DS-a2244bc0-1aa3-4ff2-946f-6856b2e72da9 from datanode e4e7b76d-7663-47f8-a739-560eaae89347 2023-02-12 03:58:33,259 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf718265a82d304a4: from storage DS-a2244bc0-1aa3-4ff2-946f-6856b2e72da9 node DatanodeRegistration(127.0.0.1:38689, datanodeUuid=e4e7b76d-7663-47f8-a739-560eaae89347, infoPort=46777, infoSecurePort=0, ipcPort=46483, storageInfo=lv=-57;cid=testClusterID;nsid=416442865;c=1676174311436), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-12 03:58:33,259 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x33b88b2285fc580e: Processing first storage report for DS-fa90b911-9a4e-4291-9d1c-1db4214946cf from datanode f058bc0e-c917-4e0e-b523-a6533f21d9b8 2023-02-12 03:58:33,259 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x33b88b2285fc580e: from storage DS-fa90b911-9a4e-4291-9d1c-1db4214946cf node DatanodeRegistration(127.0.0.1:43011, datanodeUuid=f058bc0e-c917-4e0e-b523-a6533f21d9b8, infoPort=38697, infoSecurePort=0, ipcPort=35935, storageInfo=lv=-57;cid=testClusterID;nsid=416442865;c=1676174311436), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-12 03:58:33,259 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xa2578157796d80df: Processing first storage report for DS-fd72838c-d7fc-4fe3-b355-e7b1bc90d160 from datanode d006747b-34dc-492d-8dd2-01dc9639e0d9 2023-02-12 03:58:33,259 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xa2578157796d80df: from storage DS-fd72838c-d7fc-4fe3-b355-e7b1bc90d160 node DatanodeRegistration(127.0.0.1:37239, datanodeUuid=d006747b-34dc-492d-8dd2-01dc9639e0d9, infoPort=44869, infoSecurePort=0, ipcPort=44387, storageInfo=lv=-57;cid=testClusterID;nsid=416442865;c=1676174311436), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-12 03:58:33,644 DEBUG [Listener at localhost.localdomain/46483] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5 2023-02-12 03:58:33,693 INFO [Listener at localhost.localdomain/46483] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/cluster_9ba13d3e-ed14-b1d2-0f74-8c456c2a06ab/zookeeper_0, clientPort=50368, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/cluster_9ba13d3e-ed14-b1d2-0f74-8c456c2a06ab/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/cluster_9ba13d3e-ed14-b1d2-0f74-8c456c2a06ab/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-12 03:58:33,703 INFO [Listener at localhost.localdomain/46483] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=50368 2023-02-12 03:58:33,711 INFO [Listener at localhost.localdomain/46483] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 03:58:33,712 INFO [Listener at localhost.localdomain/46483] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 03:58:34,350 INFO [Listener at localhost.localdomain/46483] util.FSUtils(479): Created version file at hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a with version=8 2023-02-12 03:58:34,350 INFO [Listener at localhost.localdomain/46483] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/hbase-staging 2023-02-12 03:58:34,589 INFO [Listener at localhost.localdomain/46483] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-12 03:58:34,965 INFO [Listener at localhost.localdomain/46483] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=18 2023-02-12 03:58:34,989 INFO [Listener at localhost.localdomain/46483] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-12 03:58:34,989 INFO [Listener at localhost.localdomain/46483] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-12 03:58:34,990 INFO [Listener at localhost.localdomain/46483] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-12 03:58:34,990 INFO [Listener at localhost.localdomain/46483] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-12 03:58:34,990 INFO [Listener at localhost.localdomain/46483] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-12 03:58:35,105 INFO [Listener at localhost.localdomain/46483] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-12 03:58:35,185 DEBUG [Listener at localhost.localdomain/46483] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-12 03:58:35,260 INFO [Listener at localhost.localdomain/46483] ipc.NettyRpcServer(120): Bind to /148.251.75.209:37305 2023-02-12 03:58:35,268 INFO [Listener at localhost.localdomain/46483] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 03:58:35,270 INFO [Listener at localhost.localdomain/46483] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 03:58:35,286 INFO [Listener at localhost.localdomain/46483] zookeeper.RecoverableZooKeeper(93): Process identifier=master:37305 connecting to ZooKeeper ensemble=127.0.0.1:50368 2023-02-12 03:58:35,317 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:373050x0, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-12 03:58:35,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:37305-0x10154d60a810000 connected 2023-02-12 03:58:35,338 DEBUG [Listener at localhost.localdomain/46483] zookeeper.ZKUtil(164): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-12 03:58:35,339 DEBUG [Listener at localhost.localdomain/46483] zookeeper.ZKUtil(164): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-12 03:58:35,343 DEBUG [Listener at localhost.localdomain/46483] zookeeper.ZKUtil(164): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-12 03:58:35,352 DEBUG [Listener at localhost.localdomain/46483] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37305 2023-02-12 03:58:35,352 DEBUG [Listener at localhost.localdomain/46483] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37305 2023-02-12 03:58:35,353 DEBUG [Listener at localhost.localdomain/46483] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37305 2023-02-12 03:58:35,353 DEBUG [Listener at localhost.localdomain/46483] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37305 2023-02-12 03:58:35,353 DEBUG [Listener at localhost.localdomain/46483] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37305 2023-02-12 03:58:35,359 INFO [Listener at localhost.localdomain/46483] master.HMaster(439): hbase.rootdir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a, hbase.cluster.distributed=false 2023-02-12 03:58:35,414 INFO [Listener at localhost.localdomain/46483] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=18 2023-02-12 03:58:35,414 INFO [Listener at localhost.localdomain/46483] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-12 03:58:35,414 INFO [Listener at localhost.localdomain/46483] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-12 03:58:35,414 INFO [Listener at localhost.localdomain/46483] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-12 03:58:35,414 INFO [Listener at localhost.localdomain/46483] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-12 03:58:35,415 INFO [Listener at localhost.localdomain/46483] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-12 03:58:35,418 INFO [Listener at localhost.localdomain/46483] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-12 03:58:35,421 INFO [Listener at localhost.localdomain/46483] ipc.NettyRpcServer(120): Bind to /148.251.75.209:42205 2023-02-12 03:58:35,423 INFO [Listener at localhost.localdomain/46483] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-12 03:58:35,428 INFO [Listener at localhost.localdomain/46483] mob.MobFileCache(124): MobFileCache disabled 2023-02-12 03:58:35,429 INFO [Listener at localhost.localdomain/46483] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 03:58:35,431 INFO [Listener at localhost.localdomain/46483] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 03:58:35,432 INFO [Listener at localhost.localdomain/46483] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42205 connecting to ZooKeeper ensemble=127.0.0.1:50368 2023-02-12 03:58:35,435 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:422050x0, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-12 03:58:35,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:42205-0x10154d60a810001 connected 2023-02-12 03:58:35,437 DEBUG [Listener at localhost.localdomain/46483] zookeeper.ZKUtil(164): regionserver:42205-0x10154d60a810001, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-12 03:58:35,438 DEBUG [Listener at localhost.localdomain/46483] zookeeper.ZKUtil(164): regionserver:42205-0x10154d60a810001, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-12 03:58:35,439 DEBUG [Listener at localhost.localdomain/46483] zookeeper.ZKUtil(164): regionserver:42205-0x10154d60a810001, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-12 03:58:35,439 DEBUG [Listener at localhost.localdomain/46483] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42205 2023-02-12 03:58:35,440 DEBUG [Listener at localhost.localdomain/46483] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42205 2023-02-12 03:58:35,440 DEBUG [Listener at localhost.localdomain/46483] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42205 2023-02-12 03:58:35,440 DEBUG [Listener at localhost.localdomain/46483] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42205 2023-02-12 03:58:35,441 DEBUG [Listener at localhost.localdomain/46483] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42205 2023-02-12 03:58:35,452 INFO [Listener at localhost.localdomain/46483] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=18 2023-02-12 03:58:35,452 INFO [Listener at localhost.localdomain/46483] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-12 03:58:35,452 INFO [Listener at localhost.localdomain/46483] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-12 03:58:35,452 INFO [Listener at localhost.localdomain/46483] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-12 03:58:35,452 INFO [Listener at localhost.localdomain/46483] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-12 03:58:35,453 INFO [Listener at localhost.localdomain/46483] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-12 03:58:35,453 INFO [Listener at localhost.localdomain/46483] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-12 03:58:35,454 INFO [Listener at localhost.localdomain/46483] ipc.NettyRpcServer(120): Bind to /148.251.75.209:35783 2023-02-12 03:58:35,455 INFO [Listener at localhost.localdomain/46483] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-12 03:58:35,456 INFO [Listener at localhost.localdomain/46483] mob.MobFileCache(124): MobFileCache disabled 2023-02-12 03:58:35,457 INFO [Listener at localhost.localdomain/46483] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 03:58:35,459 INFO [Listener at localhost.localdomain/46483] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 03:58:35,462 INFO [Listener at localhost.localdomain/46483] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35783 connecting to ZooKeeper ensemble=127.0.0.1:50368 2023-02-12 03:58:35,466 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:357830x0, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-12 03:58:35,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:35783-0x10154d60a810002 connected 2023-02-12 03:58:35,467 DEBUG [Listener at localhost.localdomain/46483] zookeeper.ZKUtil(164): regionserver:35783-0x10154d60a810002, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-12 03:58:35,468 DEBUG [Listener at localhost.localdomain/46483] zookeeper.ZKUtil(164): regionserver:35783-0x10154d60a810002, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-12 03:58:35,469 DEBUG [Listener at localhost.localdomain/46483] zookeeper.ZKUtil(164): regionserver:35783-0x10154d60a810002, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-12 03:58:35,470 DEBUG [Listener at localhost.localdomain/46483] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35783 2023-02-12 03:58:35,470 DEBUG [Listener at localhost.localdomain/46483] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35783 2023-02-12 03:58:35,470 DEBUG [Listener at localhost.localdomain/46483] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35783 2023-02-12 03:58:35,471 DEBUG [Listener at localhost.localdomain/46483] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35783 2023-02-12 03:58:35,471 DEBUG [Listener at localhost.localdomain/46483] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35783 2023-02-12 03:58:35,482 INFO [Listener at localhost.localdomain/46483] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=18 2023-02-12 03:58:35,482 INFO [Listener at localhost.localdomain/46483] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-12 03:58:35,482 INFO [Listener at localhost.localdomain/46483] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-12 03:58:35,482 INFO [Listener at localhost.localdomain/46483] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-12 03:58:35,482 INFO [Listener at localhost.localdomain/46483] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-12 03:58:35,482 INFO [Listener at localhost.localdomain/46483] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-12 03:58:35,483 INFO [Listener at localhost.localdomain/46483] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-12 03:58:35,484 INFO [Listener at localhost.localdomain/46483] ipc.NettyRpcServer(120): Bind to /148.251.75.209:35785 2023-02-12 03:58:35,484 INFO [Listener at localhost.localdomain/46483] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-12 03:58:35,485 INFO [Listener at localhost.localdomain/46483] mob.MobFileCache(124): MobFileCache disabled 2023-02-12 03:58:35,486 INFO [Listener at localhost.localdomain/46483] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 03:58:35,488 INFO [Listener at localhost.localdomain/46483] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 03:58:35,489 INFO [Listener at localhost.localdomain/46483] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35785 connecting to ZooKeeper ensemble=127.0.0.1:50368 2023-02-12 03:58:35,492 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:357850x0, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-12 03:58:35,493 DEBUG [Listener at localhost.localdomain/46483] zookeeper.ZKUtil(164): regionserver:357850x0, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-12 03:58:35,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:35785-0x10154d60a810003 connected 2023-02-12 03:58:35,494 DEBUG [Listener at localhost.localdomain/46483] zookeeper.ZKUtil(164): regionserver:35785-0x10154d60a810003, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-12 03:58:35,495 DEBUG [Listener at localhost.localdomain/46483] zookeeper.ZKUtil(164): regionserver:35785-0x10154d60a810003, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-12 03:58:35,495 DEBUG [Listener at localhost.localdomain/46483] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35785 2023-02-12 03:58:35,496 DEBUG [Listener at localhost.localdomain/46483] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35785 2023-02-12 03:58:35,496 DEBUG [Listener at localhost.localdomain/46483] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35785 2023-02-12 03:58:35,496 DEBUG [Listener at localhost.localdomain/46483] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35785 2023-02-12 03:58:35,496 DEBUG [Listener at localhost.localdomain/46483] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35785 2023-02-12 03:58:35,498 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,37305,1676174314460 2023-02-12 03:58:35,509 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-12 03:58:35,511 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,37305,1676174314460 2023-02-12 03:58:35,531 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-12 03:58:35,531 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:42205-0x10154d60a810001, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-12 03:58:35,531 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:35785-0x10154d60a810003, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-12 03:58:35,531 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:35783-0x10154d60a810002, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-12 03:58:35,531 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-12 03:58:35,532 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-12 03:58:35,533 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,37305,1676174314460 from backup master directory 2023-02-12 03:58:35,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-12 03:58:35,536 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,37305,1676174314460 2023-02-12 03:58:35,537 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-12 03:58:35,537 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-12 03:58:35,537 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase20.apache.org,37305,1676174314460 2023-02-12 03:58:35,540 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-12 03:58:35,541 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-12 03:58:35,640 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/hbase.id with ID: cbaa3b46-43ba-41e2-8d14-a7cec78c7a18 2023-02-12 03:58:35,690 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 03:58:35,705 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-12 03:58:35,750 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x0df654e0 to 127.0.0.1:50368 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 03:58:35,779 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52367b4a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 03:58:35,797 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-12 03:58:35,799 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-12 03:58:35,812 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-12 03:58:35,812 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-12 03:58:35,813 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-12 03:58:35,816 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-12 03:58:35,817 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-12 03:58:35,844 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/MasterData/data/master/store-tmp 2023-02-12 03:58:35,880 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:35,881 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-12 03:58:35,881 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-12 03:58:35,881 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-12 03:58:35,881 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-12 03:58:35,882 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-12 03:58:35,882 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-12 03:58:35,882 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676174315881Disabling compacts and flushes for region at 1676174315881Disabling writes for close at 1676174315882 (+1 ms)Writing region close event to WAL at 1676174315882Closed at 1676174315882 2023-02-12 03:58:35,883 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/MasterData/WALs/jenkins-hbase20.apache.org,37305,1676174314460 2023-02-12 03:58:35,902 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C37305%2C1676174314460, suffix=, logDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/MasterData/WALs/jenkins-hbase20.apache.org,37305,1676174314460, archiveDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/MasterData/oldWALs, maxLogs=10 2023-02-12 03:58:35,950 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43011,DS-3164a8a2-6234-417e-9b99-5b810f05d9c5,DISK] 2023-02-12 03:58:35,950 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37239,DS-293d7a97-f507-4bd5-aab1-538ffe568afe,DISK] 2023-02-12 03:58:35,950 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38689,DS-c7dfbda7-cd8b-47ec-a0b0-9d3e9f0ea3b0,DISK] 2023-02-12 03:58:35,957 DEBUG [RS-EventLoopGroup-5-1] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-12 03:58:36,011 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/MasterData/WALs/jenkins-hbase20.apache.org,37305,1676174314460/jenkins-hbase20.apache.org%2C37305%2C1676174314460.1676174315910 2023-02-12 03:58:36,011 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38689,DS-c7dfbda7-cd8b-47ec-a0b0-9d3e9f0ea3b0,DISK], DatanodeInfoWithStorage[127.0.0.1:37239,DS-293d7a97-f507-4bd5-aab1-538ffe568afe,DISK], DatanodeInfoWithStorage[127.0.0.1:43011,DS-3164a8a2-6234-417e-9b99-5b810f05d9c5,DISK]] 2023-02-12 03:58:36,012 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-12 03:58:36,012 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:36,014 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-12 03:58:36,015 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-12 03:58:36,063 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-12 03:58:36,070 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-12 03:58:36,092 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-12 03:58:36,103 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:36,108 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-12 03:58:36,110 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-12 03:58:36,122 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-12 03:58:36,126 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 03:58:36,127 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60112680, jitterRate=-0.10425126552581787}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:36,127 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676174316037Initializing all the Stores at 1676174316039 (+2 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676174316039Cleaning up temporary data from old regions at 1676174316115 (+76 ms)Cleaning up detritus from prior splits at 1676174316116 (+1 ms)Region opened successfully at 1676174316127 (+11 ms) 2023-02-12 03:58:36,128 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-12 03:58:36,144 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-12 03:58:36,144 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-12 03:58:36,146 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-12 03:58:36,148 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-02-12 03:58:36,172 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 24 msec 2023-02-12 03:58:36,172 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-12 03:58:36,193 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-12 03:58:36,198 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-12 03:58:36,218 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-12 03:58:36,221 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-12 03:58:36,222 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-12 03:58:36,226 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-12 03:58:36,229 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-12 03:58:36,231 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-12 03:58:36,233 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-12 03:58:36,233 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-12 03:58:36,244 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-12 03:58:36,247 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:35783-0x10154d60a810002, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-12 03:58:36,247 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:42205-0x10154d60a810001, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-12 03:58:36,247 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-12 03:58:36,247 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:35785-0x10154d60a810003, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-12 03:58:36,248 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-12 03:58:36,248 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase20.apache.org,37305,1676174314460, sessionid=0x10154d60a810000, setting cluster-up flag (Was=false) 2023-02-12 03:58:36,261 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-12 03:58:36,265 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-12 03:58:36,267 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,37305,1676174314460 2023-02-12 03:58:36,271 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-12 03:58:36,274 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-12 03:58:36,275 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,37305,1676174314460 2023-02-12 03:58:36,277 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp 2023-02-12 03:58:36,300 INFO [RS:2;jenkins-hbase20:35785] regionserver.HRegionServer(952): ClusterId : cbaa3b46-43ba-41e2-8d14-a7cec78c7a18 2023-02-12 03:58:36,300 INFO [RS:1;jenkins-hbase20:35783] regionserver.HRegionServer(952): ClusterId : cbaa3b46-43ba-41e2-8d14-a7cec78c7a18 2023-02-12 03:58:36,300 INFO [RS:0;jenkins-hbase20:42205] regionserver.HRegionServer(952): ClusterId : cbaa3b46-43ba-41e2-8d14-a7cec78c7a18 2023-02-12 03:58:36,305 DEBUG [RS:2;jenkins-hbase20:35785] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-12 03:58:36,305 DEBUG [RS:0;jenkins-hbase20:42205] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-12 03:58:36,305 DEBUG [RS:1;jenkins-hbase20:35783] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-12 03:58:36,309 DEBUG [RS:0;jenkins-hbase20:42205] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-12 03:58:36,310 DEBUG [RS:0;jenkins-hbase20:42205] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-12 03:58:36,309 DEBUG [RS:1;jenkins-hbase20:35783] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-12 03:58:36,309 DEBUG [RS:2;jenkins-hbase20:35785] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-12 03:58:36,310 DEBUG [RS:2;jenkins-hbase20:35785] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-12 03:58:36,310 DEBUG [RS:1;jenkins-hbase20:35783] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-12 03:58:36,312 DEBUG [RS:0;jenkins-hbase20:42205] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-12 03:58:36,312 DEBUG [RS:2;jenkins-hbase20:35785] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-12 03:58:36,312 DEBUG [RS:1;jenkins-hbase20:35783] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-12 03:58:36,314 DEBUG [RS:0;jenkins-hbase20:42205] zookeeper.ReadOnlyZKClient(139): Connect 0x36b06f89 to 127.0.0.1:50368 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 03:58:36,314 DEBUG [RS:1;jenkins-hbase20:35783] zookeeper.ReadOnlyZKClient(139): Connect 0x491252d5 to 127.0.0.1:50368 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 03:58:36,314 DEBUG [RS:2;jenkins-hbase20:35785] zookeeper.ReadOnlyZKClient(139): Connect 0x5f6bc73c to 127.0.0.1:50368 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 03:58:36,340 DEBUG [RS:0;jenkins-hbase20:42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3877872f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 03:58:36,343 DEBUG [RS:1;jenkins-hbase20:35783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@47d5a587, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 03:58:36,343 DEBUG [RS:2;jenkins-hbase20:35785] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ac22a1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 03:58:36,343 DEBUG [RS:0;jenkins-hbase20:42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c4804f9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-02-12 03:58:36,343 DEBUG [RS:1;jenkins-hbase20:35783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43ea39a5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-02-12 03:58:36,343 DEBUG [RS:2;jenkins-hbase20:35785] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5593ac4c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-02-12 03:58:36,363 DEBUG [RS:0;jenkins-hbase20:42205] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:42205 2023-02-12 03:58:36,363 DEBUG [RS:1;jenkins-hbase20:35783] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase20:35783 2023-02-12 03:58:36,365 DEBUG [RS:2;jenkins-hbase20:35785] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase20:35785 2023-02-12 03:58:36,366 INFO [RS:0;jenkins-hbase20:42205] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-12 03:58:36,366 INFO [RS:2;jenkins-hbase20:35785] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-12 03:58:36,367 INFO [RS:2;jenkins-hbase20:35785] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-12 03:58:36,366 INFO [RS:1;jenkins-hbase20:35783] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-12 03:58:36,367 DEBUG [RS:2;jenkins-hbase20:35785] regionserver.HRegionServer(1023): About to register with Master. 2023-02-12 03:58:36,367 INFO [RS:0;jenkins-hbase20:42205] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-12 03:58:36,367 INFO [RS:1;jenkins-hbase20:35783] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-12 03:58:36,367 DEBUG [RS:1;jenkins-hbase20:35783] regionserver.HRegionServer(1023): About to register with Master. 2023-02-12 03:58:36,367 DEBUG [RS:0;jenkins-hbase20:42205] regionserver.HRegionServer(1023): About to register with Master. 2023-02-12 03:58:36,370 INFO [RS:2;jenkins-hbase20:35785] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,37305,1676174314460 with isa=jenkins-hbase20.apache.org/148.251.75.209:35785, startcode=1676174315481 2023-02-12 03:58:36,370 INFO [RS:0;jenkins-hbase20:42205] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,37305,1676174314460 with isa=jenkins-hbase20.apache.org/148.251.75.209:42205, startcode=1676174315413 2023-02-12 03:58:36,370 INFO [RS:1;jenkins-hbase20:35783] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,37305,1676174314460 with isa=jenkins-hbase20.apache.org/148.251.75.209:35783, startcode=1676174315451 2023-02-12 03:58:36,387 DEBUG [RS:2;jenkins-hbase20:35785] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-12 03:58:36,387 DEBUG [RS:1;jenkins-hbase20:35783] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-12 03:58:36,387 DEBUG [RS:0;jenkins-hbase20:42205] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-12 03:58:36,434 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-12 03:58:36,441 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59177, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-12 03:58:36,441 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:58625, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-12 03:58:36,441 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42221, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-12 03:58:36,443 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-02-12 03:58:36,443 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-02-12 03:58:36,443 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-02-12 03:58:36,443 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-02-12 03:58:36,443 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-02-12 03:58:36,443 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,444 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-02-12 03:58:36,444 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,445 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676174346445 2023-02-12 03:58:36,447 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-12 03:58:36,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37305] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:36,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:36,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:36,453 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-12 03:58:36,454 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-12 03:58:36,460 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-12 03:58:36,461 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-12 03:58:36,471 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-12 03:58:36,471 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-12 03:58:36,472 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-12 03:58:36,472 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-12 03:58:36,473 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,474 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-12 03:58:36,476 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-12 03:58:36,476 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-12 03:58:36,478 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-12 03:58:36,479 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-12 03:58:36,481 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1676174316480,5,FailOnTimeoutGroup] 2023-02-12 03:58:36,481 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1676174316481,5,FailOnTimeoutGroup] 2023-02-12 03:58:36,481 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,481 DEBUG [RS:2;jenkins-hbase20:35785] regionserver.HRegionServer(2829): Master is not running yet 2023-02-12 03:58:36,482 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-12 03:58:36,482 WARN [RS:2;jenkins-hbase20:35785] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-12 03:58:36,481 DEBUG [RS:1;jenkins-hbase20:35783] regionserver.HRegionServer(2829): Master is not running yet 2023-02-12 03:58:36,481 DEBUG [RS:0;jenkins-hbase20:42205] regionserver.HRegionServer(2829): Master is not running yet 2023-02-12 03:58:36,482 WARN [RS:1;jenkins-hbase20:35783] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-12 03:58:36,482 WARN [RS:0;jenkins-hbase20:42205] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-12 03:58:36,483 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,483 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,502 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-12 03:58:36,503 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-12 03:58:36,504 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a 2023-02-12 03:58:36,528 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:36,531 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-12 03:58:36,534 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/info 2023-02-12 03:58:36,535 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-12 03:58:36,536 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:36,536 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-12 03:58:36,539 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/rep_barrier 2023-02-12 03:58:36,540 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-12 03:58:36,541 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:36,541 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-12 03:58:36,543 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/table 2023-02-12 03:58:36,544 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-12 03:58:36,545 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:36,547 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740 2023-02-12 03:58:36,549 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740 2023-02-12 03:58:36,554 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-12 03:58:36,556 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-12 03:58:36,561 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 03:58:36,562 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61859532, jitterRate=-0.0782211422920227}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-12 03:58:36,562 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676174316528Initializing all the Stores at 1676174316530 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676174316530Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676174316530Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676174316530Cleaning up temporary data from old regions at 1676174316552 (+22 ms)Cleaning up detritus from prior splits at 1676174316552Region opened successfully at 1676174316562 (+10 ms) 2023-02-12 03:58:36,562 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-12 03:58:36,562 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-12 03:58:36,562 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-12 03:58:36,562 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-12 03:58:36,562 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-12 03:58:36,563 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-12 03:58:36,563 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676174316562Disabling compacts and flushes for region at 1676174316562Disabling writes for close at 1676174316562Writing region close event to WAL at 1676174316563 (+1 ms)Closed at 1676174316563 2023-02-12 03:58:36,569 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-12 03:58:36,569 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-12 03:58:36,576 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-12 03:58:36,583 INFO [RS:2;jenkins-hbase20:35785] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,37305,1676174314460 with isa=jenkins-hbase20.apache.org/148.251.75.209:35785, startcode=1676174315481 2023-02-12 03:58:36,584 INFO [RS:0;jenkins-hbase20:42205] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,37305,1676174314460 with isa=jenkins-hbase20.apache.org/148.251.75.209:42205, startcode=1676174315413 2023-02-12 03:58:36,584 INFO [RS:1;jenkins-hbase20:35783] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,37305,1676174314460 with isa=jenkins-hbase20.apache.org/148.251.75.209:35783, startcode=1676174315451 2023-02-12 03:58:36,588 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-12 03:58:36,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37305] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:36,591 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37305] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:36,592 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:36,595 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-12 03:58:36,599 DEBUG [RS:1;jenkins-hbase20:35783] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a 2023-02-12 03:58:36,599 DEBUG [RS:2;jenkins-hbase20:35785] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a 2023-02-12 03:58:36,599 DEBUG [RS:0;jenkins-hbase20:42205] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a 2023-02-12 03:58:36,599 DEBUG [RS:2;jenkins-hbase20:35785] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:46179 2023-02-12 03:58:36,599 DEBUG [RS:1;jenkins-hbase20:35783] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:46179 2023-02-12 03:58:36,599 DEBUG [RS:2;jenkins-hbase20:35785] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-12 03:58:36,599 DEBUG [RS:0;jenkins-hbase20:42205] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:46179 2023-02-12 03:58:36,599 DEBUG [RS:1;jenkins-hbase20:35783] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-12 03:58:36,599 DEBUG [RS:0;jenkins-hbase20:42205] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-12 03:58:36,606 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-12 03:58:36,606 DEBUG [RS:2;jenkins-hbase20:35785] zookeeper.ZKUtil(162): regionserver:35785-0x10154d60a810003, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:36,606 DEBUG [RS:1;jenkins-hbase20:35783] zookeeper.ZKUtil(162): regionserver:35783-0x10154d60a810002, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:36,606 WARN [RS:2;jenkins-hbase20:35785] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-12 03:58:36,606 DEBUG [RS:0;jenkins-hbase20:42205] zookeeper.ZKUtil(162): regionserver:42205-0x10154d60a810001, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:36,607 INFO [RS:2;jenkins-hbase20:35785] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-12 03:58:36,606 WARN [RS:1;jenkins-hbase20:35783] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-12 03:58:36,607 WARN [RS:0;jenkins-hbase20:42205] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-12 03:58:36,608 DEBUG [RS:2;jenkins-hbase20:35785] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/WALs/jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:36,608 INFO [RS:0;jenkins-hbase20:42205] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-12 03:58:36,607 INFO [RS:1;jenkins-hbase20:35783] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-12 03:58:36,608 DEBUG [RS:0;jenkins-hbase20:42205] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/WALs/jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:36,608 DEBUG [RS:1;jenkins-hbase20:35783] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/WALs/jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:36,611 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,42205,1676174315413] 2023-02-12 03:58:36,612 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,35785,1676174315481] 2023-02-12 03:58:36,612 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,35783,1676174315451] 2023-02-12 03:58:36,622 DEBUG [RS:2;jenkins-hbase20:35785] zookeeper.ZKUtil(162): regionserver:35785-0x10154d60a810003, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:36,622 DEBUG [RS:1;jenkins-hbase20:35783] zookeeper.ZKUtil(162): regionserver:35783-0x10154d60a810002, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:36,623 DEBUG [RS:0;jenkins-hbase20:42205] zookeeper.ZKUtil(162): regionserver:42205-0x10154d60a810001, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:36,623 DEBUG [RS:2;jenkins-hbase20:35785] zookeeper.ZKUtil(162): regionserver:35785-0x10154d60a810003, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:36,623 DEBUG [RS:1;jenkins-hbase20:35783] zookeeper.ZKUtil(162): regionserver:35783-0x10154d60a810002, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:36,623 DEBUG [RS:0;jenkins-hbase20:42205] zookeeper.ZKUtil(162): regionserver:42205-0x10154d60a810001, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:36,623 DEBUG [RS:2;jenkins-hbase20:35785] zookeeper.ZKUtil(162): regionserver:35785-0x10154d60a810003, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:36,624 DEBUG [RS:1;jenkins-hbase20:35783] zookeeper.ZKUtil(162): regionserver:35783-0x10154d60a810002, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:36,624 DEBUG [RS:0;jenkins-hbase20:42205] zookeeper.ZKUtil(162): regionserver:42205-0x10154d60a810001, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:36,634 DEBUG [RS:2;jenkins-hbase20:35785] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-12 03:58:36,634 DEBUG [RS:1;jenkins-hbase20:35783] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-12 03:58:36,634 DEBUG [RS:0;jenkins-hbase20:42205] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-12 03:58:36,643 INFO [RS:1;jenkins-hbase20:35783] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-12 03:58:36,643 INFO [RS:0;jenkins-hbase20:42205] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-12 03:58:36,643 INFO [RS:2;jenkins-hbase20:35785] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-12 03:58:36,664 INFO [RS:1;jenkins-hbase20:35783] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-12 03:58:36,664 INFO [RS:2;jenkins-hbase20:35785] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-12 03:58:36,664 INFO [RS:0;jenkins-hbase20:42205] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-12 03:58:36,668 INFO [RS:2;jenkins-hbase20:35785] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-12 03:58:36,668 INFO [RS:1;jenkins-hbase20:35783] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-12 03:58:36,668 INFO [RS:0;jenkins-hbase20:42205] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-12 03:58:36,669 INFO [RS:1;jenkins-hbase20:35783] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,669 INFO [RS:2;jenkins-hbase20:35785] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,669 INFO [RS:0;jenkins-hbase20:42205] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,670 INFO [RS:0;jenkins-hbase20:42205] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-12 03:58:36,670 INFO [RS:2;jenkins-hbase20:35785] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-12 03:58:36,670 INFO [RS:1;jenkins-hbase20:35783] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-12 03:58:36,676 INFO [RS:2;jenkins-hbase20:35785] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,676 INFO [RS:1;jenkins-hbase20:35783] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,676 INFO [RS:0;jenkins-hbase20:42205] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,677 DEBUG [RS:2;jenkins-hbase20:35785] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,677 DEBUG [RS:1;jenkins-hbase20:35783] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,677 DEBUG [RS:2;jenkins-hbase20:35785] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,677 DEBUG [RS:1;jenkins-hbase20:35783] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,677 DEBUG [RS:2;jenkins-hbase20:35785] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,677 DEBUG [RS:1;jenkins-hbase20:35783] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,677 DEBUG [RS:0;jenkins-hbase20:42205] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,677 DEBUG [RS:1;jenkins-hbase20:35783] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,677 DEBUG [RS:0;jenkins-hbase20:42205] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,677 DEBUG [RS:1;jenkins-hbase20:35783] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,678 DEBUG [RS:0;jenkins-hbase20:42205] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,678 DEBUG [RS:1;jenkins-hbase20:35783] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-02-12 03:58:36,678 DEBUG [RS:0;jenkins-hbase20:42205] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,678 DEBUG [RS:1;jenkins-hbase20:35783] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,678 DEBUG [RS:0;jenkins-hbase20:42205] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,677 DEBUG [RS:2;jenkins-hbase20:35785] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,678 DEBUG [RS:0;jenkins-hbase20:42205] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-02-12 03:58:36,678 DEBUG [RS:2;jenkins-hbase20:35785] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,678 DEBUG [RS:0;jenkins-hbase20:42205] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,678 DEBUG [RS:2;jenkins-hbase20:35785] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-02-12 03:58:36,678 DEBUG [RS:0;jenkins-hbase20:42205] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,678 DEBUG [RS:2;jenkins-hbase20:35785] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,678 DEBUG [RS:1;jenkins-hbase20:35783] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,678 DEBUG [RS:2;jenkins-hbase20:35785] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,678 DEBUG [RS:1;jenkins-hbase20:35783] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,678 DEBUG [RS:2;jenkins-hbase20:35785] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,678 DEBUG [RS:1;jenkins-hbase20:35783] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,678 DEBUG [RS:2;jenkins-hbase20:35785] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,678 DEBUG [RS:0;jenkins-hbase20:42205] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,678 DEBUG [RS:0;jenkins-hbase20:42205] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-02-12 03:58:36,680 INFO [RS:1;jenkins-hbase20:35783] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,680 INFO [RS:2;jenkins-hbase20:35785] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,680 INFO [RS:1;jenkins-hbase20:35783] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,680 INFO [RS:2;jenkins-hbase20:35785] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,680 INFO [RS:1;jenkins-hbase20:35783] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,680 INFO [RS:2;jenkins-hbase20:35785] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,683 INFO [RS:0;jenkins-hbase20:42205] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,683 INFO [RS:0;jenkins-hbase20:42205] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,683 INFO [RS:0;jenkins-hbase20:42205] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,695 INFO [RS:1;jenkins-hbase20:35783] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-12 03:58:36,696 INFO [RS:2;jenkins-hbase20:35785] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-12 03:58:36,698 INFO [RS:2;jenkins-hbase20:35785] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,35785,1676174315481-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,698 INFO [RS:1;jenkins-hbase20:35783] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,35783,1676174315451-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,699 INFO [RS:0;jenkins-hbase20:42205] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-12 03:58:36,699 INFO [RS:0;jenkins-hbase20:42205] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,42205,1676174315413-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:36,713 INFO [RS:2;jenkins-hbase20:35785] regionserver.Replication(203): jenkins-hbase20.apache.org,35785,1676174315481 started 2023-02-12 03:58:36,713 INFO [RS:0;jenkins-hbase20:42205] regionserver.Replication(203): jenkins-hbase20.apache.org,42205,1676174315413 started 2023-02-12 03:58:36,713 INFO [RS:0;jenkins-hbase20:42205] regionserver.HRegionServer(1638): Serving as jenkins-hbase20.apache.org,42205,1676174315413, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:42205, sessionid=0x10154d60a810001 2023-02-12 03:58:36,713 INFO [RS:2;jenkins-hbase20:35785] regionserver.HRegionServer(1638): Serving as jenkins-hbase20.apache.org,35785,1676174315481, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:35785, sessionid=0x10154d60a810003 2023-02-12 03:58:36,713 DEBUG [RS:0;jenkins-hbase20:42205] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-12 03:58:36,713 DEBUG [RS:2;jenkins-hbase20:35785] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-12 03:58:36,713 DEBUG [RS:0;jenkins-hbase20:42205] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:36,713 DEBUG [RS:2;jenkins-hbase20:35785] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:36,714 DEBUG [RS:0;jenkins-hbase20:42205] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,42205,1676174315413' 2023-02-12 03:58:36,714 DEBUG [RS:2;jenkins-hbase20:35785] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,35785,1676174315481' 2023-02-12 03:58:36,715 DEBUG [RS:2;jenkins-hbase20:35785] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-12 03:58:36,715 DEBUG [RS:0;jenkins-hbase20:42205] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-12 03:58:36,715 DEBUG [RS:0;jenkins-hbase20:42205] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-12 03:58:36,715 DEBUG [RS:2;jenkins-hbase20:35785] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-12 03:58:36,716 DEBUG [RS:0;jenkins-hbase20:42205] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-12 03:58:36,716 DEBUG [RS:2;jenkins-hbase20:35785] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-12 03:58:36,716 DEBUG [RS:0;jenkins-hbase20:42205] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-12 03:58:36,716 DEBUG [RS:2;jenkins-hbase20:35785] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-12 03:58:36,716 DEBUG [RS:0;jenkins-hbase20:42205] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:36,716 DEBUG [RS:2;jenkins-hbase20:35785] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:36,717 DEBUG [RS:2;jenkins-hbase20:35785] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,35785,1676174315481' 2023-02-12 03:58:36,717 DEBUG [RS:0;jenkins-hbase20:42205] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,42205,1676174315413' 2023-02-12 03:58:36,717 DEBUG [RS:0;jenkins-hbase20:42205] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-12 03:58:36,717 DEBUG [RS:2;jenkins-hbase20:35785] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-12 03:58:36,717 DEBUG [RS:0;jenkins-hbase20:42205] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-12 03:58:36,717 DEBUG [RS:2;jenkins-hbase20:35785] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-12 03:58:36,717 INFO [RS:1;jenkins-hbase20:35783] regionserver.Replication(203): jenkins-hbase20.apache.org,35783,1676174315451 started 2023-02-12 03:58:36,718 INFO [RS:1;jenkins-hbase20:35783] regionserver.HRegionServer(1638): Serving as jenkins-hbase20.apache.org,35783,1676174315451, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:35783, sessionid=0x10154d60a810002 2023-02-12 03:58:36,718 DEBUG [RS:0;jenkins-hbase20:42205] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-12 03:58:36,718 DEBUG [RS:1;jenkins-hbase20:35783] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-12 03:58:36,718 DEBUG [RS:2;jenkins-hbase20:35785] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-12 03:58:36,718 DEBUG [RS:1;jenkins-hbase20:35783] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:36,718 INFO [RS:0;jenkins-hbase20:42205] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-12 03:58:36,718 DEBUG [RS:1;jenkins-hbase20:35783] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,35783,1676174315451' 2023-02-12 03:58:36,719 DEBUG [RS:1;jenkins-hbase20:35783] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-12 03:58:36,718 INFO [RS:2;jenkins-hbase20:35785] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-12 03:58:36,719 INFO [RS:0;jenkins-hbase20:42205] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-12 03:58:36,719 INFO [RS:2;jenkins-hbase20:35785] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-12 03:58:36,719 DEBUG [RS:1;jenkins-hbase20:35783] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-12 03:58:36,720 DEBUG [RS:1;jenkins-hbase20:35783] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-12 03:58:36,720 DEBUG [RS:1;jenkins-hbase20:35783] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-12 03:58:36,720 DEBUG [RS:1;jenkins-hbase20:35783] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:36,720 DEBUG [RS:1;jenkins-hbase20:35783] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,35783,1676174315451' 2023-02-12 03:58:36,720 DEBUG [RS:1;jenkins-hbase20:35783] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-12 03:58:36,720 DEBUG [RS:1;jenkins-hbase20:35783] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-12 03:58:36,721 DEBUG [RS:1;jenkins-hbase20:35783] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-12 03:58:36,721 INFO [RS:1;jenkins-hbase20:35783] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-12 03:58:36,721 INFO [RS:1;jenkins-hbase20:35783] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-12 03:58:36,747 DEBUG [jenkins-hbase20:37305] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-12 03:58:36,750 DEBUG [jenkins-hbase20:37305] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-02-12 03:58:36,756 DEBUG [jenkins-hbase20:37305] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-12 03:58:36,756 DEBUG [jenkins-hbase20:37305] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-12 03:58:36,756 DEBUG [jenkins-hbase20:37305] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-12 03:58:36,756 DEBUG [jenkins-hbase20:37305] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-12 03:58:36,758 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,35783,1676174315451, state=OPENING 2023-02-12 03:58:36,764 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-12 03:58:36,765 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-12 03:58:36,766 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-12 03:58:36,769 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:36,830 INFO [RS:0;jenkins-hbase20:42205] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C42205%2C1676174315413, suffix=, logDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/WALs/jenkins-hbase20.apache.org,42205,1676174315413, archiveDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/oldWALs, maxLogs=32 2023-02-12 03:58:36,831 INFO [RS:1;jenkins-hbase20:35783] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C35783%2C1676174315451, suffix=, logDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/WALs/jenkins-hbase20.apache.org,35783,1676174315451, archiveDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/oldWALs, maxLogs=32 2023-02-12 03:58:36,830 INFO [RS:2;jenkins-hbase20:35785] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C35785%2C1676174315481, suffix=, logDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/WALs/jenkins-hbase20.apache.org,35785,1676174315481, archiveDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/oldWALs, maxLogs=32 2023-02-12 03:58:36,857 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43011,DS-3164a8a2-6234-417e-9b99-5b810f05d9c5,DISK] 2023-02-12 03:58:36,857 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38689,DS-c7dfbda7-cd8b-47ec-a0b0-9d3e9f0ea3b0,DISK] 2023-02-12 03:58:36,857 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38689,DS-c7dfbda7-cd8b-47ec-a0b0-9d3e9f0ea3b0,DISK] 2023-02-12 03:58:36,859 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37239,DS-293d7a97-f507-4bd5-aab1-538ffe568afe,DISK] 2023-02-12 03:58:36,859 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37239,DS-293d7a97-f507-4bd5-aab1-538ffe568afe,DISK] 2023-02-12 03:58:36,859 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43011,DS-3164a8a2-6234-417e-9b99-5b810f05d9c5,DISK] 2023-02-12 03:58:36,859 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43011,DS-3164a8a2-6234-417e-9b99-5b810f05d9c5,DISK] 2023-02-12 03:58:36,859 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37239,DS-293d7a97-f507-4bd5-aab1-538ffe568afe,DISK] 2023-02-12 03:58:36,860 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38689,DS-c7dfbda7-cd8b-47ec-a0b0-9d3e9f0ea3b0,DISK] 2023-02-12 03:58:36,873 INFO [RS:2;jenkins-hbase20:35785] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/WALs/jenkins-hbase20.apache.org,35785,1676174315481/jenkins-hbase20.apache.org%2C35785%2C1676174315481.1676174316836 2023-02-12 03:58:36,873 INFO [RS:0;jenkins-hbase20:42205] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/WALs/jenkins-hbase20.apache.org,42205,1676174315413/jenkins-hbase20.apache.org%2C42205%2C1676174315413.1676174316836 2023-02-12 03:58:36,873 INFO [RS:1;jenkins-hbase20:35783] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/WALs/jenkins-hbase20.apache.org,35783,1676174315451/jenkins-hbase20.apache.org%2C35783%2C1676174315451.1676174316836 2023-02-12 03:58:36,873 DEBUG [RS:2;jenkins-hbase20:35785] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38689,DS-c7dfbda7-cd8b-47ec-a0b0-9d3e9f0ea3b0,DISK], DatanodeInfoWithStorage[127.0.0.1:37239,DS-293d7a97-f507-4bd5-aab1-538ffe568afe,DISK], DatanodeInfoWithStorage[127.0.0.1:43011,DS-3164a8a2-6234-417e-9b99-5b810f05d9c5,DISK]] 2023-02-12 03:58:36,873 DEBUG [RS:0;jenkins-hbase20:42205] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43011,DS-3164a8a2-6234-417e-9b99-5b810f05d9c5,DISK], DatanodeInfoWithStorage[127.0.0.1:37239,DS-293d7a97-f507-4bd5-aab1-538ffe568afe,DISK], DatanodeInfoWithStorage[127.0.0.1:38689,DS-c7dfbda7-cd8b-47ec-a0b0-9d3e9f0ea3b0,DISK]] 2023-02-12 03:58:36,873 DEBUG [RS:1;jenkins-hbase20:35783] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38689,DS-c7dfbda7-cd8b-47ec-a0b0-9d3e9f0ea3b0,DISK], DatanodeInfoWithStorage[127.0.0.1:37239,DS-293d7a97-f507-4bd5-aab1-538ffe568afe,DISK], DatanodeInfoWithStorage[127.0.0.1:43011,DS-3164a8a2-6234-417e-9b99-5b810f05d9c5,DISK]] 2023-02-12 03:58:36,953 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:36,956 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-12 03:58:36,959 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47284, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-12 03:58:36,969 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-12 03:58:36,970 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-12 03:58:36,974 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C35783%2C1676174315451.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/WALs/jenkins-hbase20.apache.org,35783,1676174315451, archiveDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/oldWALs, maxLogs=32 2023-02-12 03:58:36,993 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37239,DS-293d7a97-f507-4bd5-aab1-538ffe568afe,DISK] 2023-02-12 03:58:36,994 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43011,DS-3164a8a2-6234-417e-9b99-5b810f05d9c5,DISK] 2023-02-12 03:58:36,998 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38689,DS-c7dfbda7-cd8b-47ec-a0b0-9d3e9f0ea3b0,DISK] 2023-02-12 03:58:37,011 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/WALs/jenkins-hbase20.apache.org,35783,1676174315451/jenkins-hbase20.apache.org%2C35783%2C1676174315451.meta.1676174316975.meta 2023-02-12 03:58:37,014 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37239,DS-293d7a97-f507-4bd5-aab1-538ffe568afe,DISK], DatanodeInfoWithStorage[127.0.0.1:43011,DS-3164a8a2-6234-417e-9b99-5b810f05d9c5,DISK], DatanodeInfoWithStorage[127.0.0.1:38689,DS-c7dfbda7-cd8b-47ec-a0b0-9d3e9f0ea3b0,DISK]] 2023-02-12 03:58:37,014 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-12 03:58:37,016 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-12 03:58:37,031 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-12 03:58:37,035 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-12 03:58:37,040 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-12 03:58:37,040 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:37,040 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-12 03:58:37,040 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-12 03:58:37,043 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-12 03:58:37,045 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/info 2023-02-12 03:58:37,045 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/info 2023-02-12 03:58:37,046 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-12 03:58:37,047 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:37,047 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-12 03:58:37,049 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/rep_barrier 2023-02-12 03:58:37,049 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/rep_barrier 2023-02-12 03:58:37,050 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-12 03:58:37,051 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:37,051 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-12 03:58:37,052 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/table 2023-02-12 03:58:37,052 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/table 2023-02-12 03:58:37,053 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-12 03:58:37,054 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:37,056 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740 2023-02-12 03:58:37,058 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740 2023-02-12 03:58:37,062 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-12 03:58:37,064 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-12 03:58:37,066 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67277048, jitterRate=0.0025061368942260742}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-12 03:58:37,066 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676174317040Writing region info on filesystem at 1676174317041 (+1 ms)Initializing all the Stores at 1676174317042 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676174317042Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676174317042Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676174317043 (+1 ms)Cleaning up temporary data from old regions at 1676174317060 (+17 ms)Cleaning up detritus from prior splits at 1676174317061 (+1 ms)Running coprocessor post-open hooks at 1676174317066 (+5 ms)Region opened successfully at 1676174317066 2023-02-12 03:58:37,076 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676174316945 2023-02-12 03:58:37,094 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-12 03:58:37,095 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-12 03:58:37,095 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,35783,1676174315451, state=OPEN 2023-02-12 03:58:37,097 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-12 03:58:37,097 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-12 03:58:37,102 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-12 03:58:37,102 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,35783,1676174315451 in 328 msec 2023-02-12 03:58:37,109 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-12 03:58:37,110 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 528 msec 2023-02-12 03:58:37,114 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 761 msec 2023-02-12 03:58:37,114 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676174315528, completionTime=-1 2023-02-12 03:58:37,115 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-12 03:58:37,115 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-12 03:58:37,175 DEBUG [hconnection-0x7a990b1c-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:37,179 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47300, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:37,195 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-12 03:58:37,195 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676174377195 2023-02-12 03:58:37,196 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676174437196 2023-02-12 03:58:37,196 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 80 msec 2023-02-12 03:58:37,236 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,37305,1676174314460-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:37,236 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,37305,1676174314460-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:37,236 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,37305,1676174314460-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:37,238 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:37305, period=300000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:37,239 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-12 03:58:37,247 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-12 03:58:37,254 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-12 03:58:37,255 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-12 03:58:37,265 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-12 03:58:37,267 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-12 03:58:37,271 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-12 03:58:37,297 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/hbase/namespace/d852011c2bc11fbb7a1e9bd27375aba0 2023-02-12 03:58:37,299 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/hbase/namespace/d852011c2bc11fbb7a1e9bd27375aba0 empty. 2023-02-12 03:58:37,300 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/hbase/namespace/d852011c2bc11fbb7a1e9bd27375aba0 2023-02-12 03:58:37,300 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-12 03:58:37,346 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-12 03:58:37,348 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => d852011c2bc11fbb7a1e9bd27375aba0, NAME => 'hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp 2023-02-12 03:58:37,369 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:37,369 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing d852011c2bc11fbb7a1e9bd27375aba0, disabling compactions & flushes 2023-02-12 03:58:37,369 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0. 2023-02-12 03:58:37,369 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0. 2023-02-12 03:58:37,369 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0. after waiting 0 ms 2023-02-12 03:58:37,369 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0. 2023-02-12 03:58:37,370 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0. 2023-02-12 03:58:37,370 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for d852011c2bc11fbb7a1e9bd27375aba0: Waiting for close lock at 1676174317369Disabling compacts and flushes for region at 1676174317369Disabling writes for close at 1676174317369Writing region close event to WAL at 1676174317370 (+1 ms)Closed at 1676174317370 2023-02-12 03:58:37,375 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-12 03:58:37,388 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676174317377"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174317377"}]},"ts":"1676174317377"} 2023-02-12 03:58:37,413 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-12 03:58:37,417 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-12 03:58:37,424 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174317417"}]},"ts":"1676174317417"} 2023-02-12 03:58:37,429 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-12 03:58:37,433 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-02-12 03:58:37,435 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-12 03:58:37,435 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-12 03:58:37,435 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-12 03:58:37,435 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-12 03:58:37,438 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=d852011c2bc11fbb7a1e9bd27375aba0, ASSIGN}] 2023-02-12 03:58:37,442 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=d852011c2bc11fbb7a1e9bd27375aba0, ASSIGN 2023-02-12 03:58:37,444 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=d852011c2bc11fbb7a1e9bd27375aba0, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,35783,1676174315451; forceNewPlan=false, retain=false 2023-02-12 03:58:37,598 INFO [jenkins-hbase20:37305] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-12 03:58:37,599 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=d852011c2bc11fbb7a1e9bd27375aba0, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:37,600 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676174317599"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174317599"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174317599"}]},"ts":"1676174317599"} 2023-02-12 03:58:37,604 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure d852011c2bc11fbb7a1e9bd27375aba0, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:37,764 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0. 2023-02-12 03:58:37,764 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d852011c2bc11fbb7a1e9bd27375aba0, NAME => 'hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0.', STARTKEY => '', ENDKEY => ''} 2023-02-12 03:58:37,765 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace d852011c2bc11fbb7a1e9bd27375aba0 2023-02-12 03:58:37,765 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:37,765 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for d852011c2bc11fbb7a1e9bd27375aba0 2023-02-12 03:58:37,765 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for d852011c2bc11fbb7a1e9bd27375aba0 2023-02-12 03:58:37,767 INFO [StoreOpener-d852011c2bc11fbb7a1e9bd27375aba0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region d852011c2bc11fbb7a1e9bd27375aba0 2023-02-12 03:58:37,769 DEBUG [StoreOpener-d852011c2bc11fbb7a1e9bd27375aba0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/namespace/d852011c2bc11fbb7a1e9bd27375aba0/info 2023-02-12 03:58:37,769 DEBUG [StoreOpener-d852011c2bc11fbb7a1e9bd27375aba0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/namespace/d852011c2bc11fbb7a1e9bd27375aba0/info 2023-02-12 03:58:37,770 INFO [StoreOpener-d852011c2bc11fbb7a1e9bd27375aba0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d852011c2bc11fbb7a1e9bd27375aba0 columnFamilyName info 2023-02-12 03:58:37,770 INFO [StoreOpener-d852011c2bc11fbb7a1e9bd27375aba0-1] regionserver.HStore(310): Store=d852011c2bc11fbb7a1e9bd27375aba0/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:37,772 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/namespace/d852011c2bc11fbb7a1e9bd27375aba0 2023-02-12 03:58:37,774 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/namespace/d852011c2bc11fbb7a1e9bd27375aba0 2023-02-12 03:58:37,777 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for d852011c2bc11fbb7a1e9bd27375aba0 2023-02-12 03:58:37,781 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/namespace/d852011c2bc11fbb7a1e9bd27375aba0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 03:58:37,782 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened d852011c2bc11fbb7a1e9bd27375aba0; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72058436, jitterRate=0.07375437021255493}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:37,783 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for d852011c2bc11fbb7a1e9bd27375aba0: Running coprocessor pre-open hook at 1676174317765Writing region info on filesystem at 1676174317765Initializing all the Stores at 1676174317767 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676174317767Cleaning up temporary data from old regions at 1676174317774 (+7 ms)Cleaning up detritus from prior splits at 1676174317775 (+1 ms)Running coprocessor post-open hooks at 1676174317783 (+8 ms)Region opened successfully at 1676174317783 2023-02-12 03:58:37,785 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0., pid=6, masterSystemTime=1676174317758 2023-02-12 03:58:37,789 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0. 2023-02-12 03:58:37,789 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0. 2023-02-12 03:58:37,790 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=d852011c2bc11fbb7a1e9bd27375aba0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:37,791 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676174317789"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174317789"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174317789"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174317789"}]},"ts":"1676174317789"} 2023-02-12 03:58:37,798 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-12 03:58:37,798 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure d852011c2bc11fbb7a1e9bd27375aba0, server=jenkins-hbase20.apache.org,35783,1676174315451 in 190 msec 2023-02-12 03:58:37,804 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-12 03:58:37,805 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=d852011c2bc11fbb7a1e9bd27375aba0, ASSIGN in 360 msec 2023-02-12 03:58:37,807 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-12 03:58:37,807 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174317807"}]},"ts":"1676174317807"} 2023-02-12 03:58:37,810 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-12 03:58:37,813 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-12 03:58:37,816 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 556 msec 2023-02-12 03:58:37,868 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-12 03:58:37,869 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-12 03:58:37,869 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-12 03:58:37,905 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-12 03:58:37,922 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-12 03:58:37,929 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 32 msec 2023-02-12 03:58:37,939 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-12 03:58:37,951 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-12 03:58:37,956 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-02-12 03:58:37,970 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-12 03:58:37,971 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-12 03:58:37,972 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.434sec 2023-02-12 03:58:37,974 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-12 03:58:37,975 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-12 03:58:37,976 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-12 03:58:37,977 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,37305,1676174314460-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-12 03:58:37,978 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,37305,1676174314460-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-12 03:58:37,986 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-12 03:58:38,007 DEBUG [Listener at localhost.localdomain/46483] zookeeper.ReadOnlyZKClient(139): Connect 0x0d193e18 to 127.0.0.1:50368 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 03:58:38,011 DEBUG [Listener at localhost.localdomain/46483] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bd70031, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 03:58:38,025 DEBUG [hconnection-0x51e75f37-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:38,036 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47304, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:38,046 INFO [Listener at localhost.localdomain/46483] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,37305,1676174314460 2023-02-12 03:58:38,076 INFO [Listener at localhost.localdomain/46483] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=358, OpenFileDescriptor=686, MaxFileDescriptor=60000, SystemLoadAverage=143, ProcessCount=164, AvailableMemoryMB=5017 2023-02-12 03:58:38,088 DEBUG [Listener at localhost.localdomain/46483] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-12 03:58:38,091 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49656, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-12 03:58:38,102 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$4(2102): Client=jenkins//148.251.75.209 create 'testRestoreSnapshot-1676174318078', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-12 03:58:38,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:38,108 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676174318078 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-12 03:58:38,110 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676174318078 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-12 03:58:38,112 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "testRestoreSnapshot-1676174318078" procId is: 9 2023-02-12 03:58:38,118 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:38,118 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:38,118 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:38,118 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:38,121 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1 empty. 2023-02-12 03:58:38,121 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:38,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-12 03:58:38,121 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:38,122 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608 empty. 2023-02-12 03:58:38,124 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c empty. 2023-02-12 03:58:38,125 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45 empty. 2023-02-12 03:58:38,125 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:38,125 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a empty. 2023-02-12 03:58:38,126 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76 empty. 2023-02-12 03:58:38,126 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:38,129 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:38,129 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:38,131 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:38,131 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:38,131 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676174318078 regions 2023-02-12 03:58:38,162 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/.tabledesc/.tableinfo.0000000001 2023-02-12 03:58:38,165 INFO [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-1] regionserver.HRegion(7671): creating {ENCODED => ecda59e01589f4f279b19a5420a1e608, NAME => 'testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676174318078', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp 2023-02-12 03:58:38,165 INFO [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-2] regionserver.HRegion(7671): creating {ENCODED => fcd2ed887eb9f47c07236428879982c1, NAME => 'testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676174318078', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp 2023-02-12 03:58:38,165 INFO [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-0] regionserver.HRegion(7671): creating {ENCODED => cc0e0b01138d4eae5df1e5641b484f45, NAME => 'testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676174318078', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp 2023-02-12 03:58:38,222 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:38,222 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:38,224 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-2] regionserver.HRegion(1603): Closing fcd2ed887eb9f47c07236428879982c1, disabling compactions & flushes 2023-02-12 03:58:38,224 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-1] regionserver.HRegion(1603): Closing ecda59e01589f4f279b19a5420a1e608, disabling compactions & flushes 2023-02-12 03:58:38,224 INFO [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:38,225 INFO [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:38,225 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:38,225 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. after waiting 0 ms 2023-02-12 03:58:38,225 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:38,225 INFO [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:38,225 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-2] regionserver.HRegion(1557): Region close journal for fcd2ed887eb9f47c07236428879982c1: Waiting for close lock at 1676174318224Disabling compacts and flushes for region at 1676174318224Disabling writes for close at 1676174318225 (+1 ms)Writing region close event to WAL at 1676174318225Closed at 1676174318225 2023-02-12 03:58:38,225 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:38,226 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. after waiting 0 ms 2023-02-12 03:58:38,226 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:38,226 INFO [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-2] regionserver.HRegion(7671): creating {ENCODED => 609b1a331ab507675f39b4bdbe7a382c, NAME => 'testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676174318078', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp 2023-02-12 03:58:38,226 INFO [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:38,226 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-1] regionserver.HRegion(1557): Region close journal for ecda59e01589f4f279b19a5420a1e608: Waiting for close lock at 1676174318224Disabling compacts and flushes for region at 1676174318224Disabling writes for close at 1676174318226 (+2 ms)Writing region close event to WAL at 1676174318226Closed at 1676174318226 2023-02-12 03:58:38,226 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:38,229 INFO [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-1] regionserver.HRegion(7671): creating {ENCODED => e019e3f4c00b3da154d59b4085911d76, NAME => 'testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676174318078', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp 2023-02-12 03:58:38,230 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-0] regionserver.HRegion(1603): Closing cc0e0b01138d4eae5df1e5641b484f45, disabling compactions & flushes 2023-02-12 03:58:38,231 INFO [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:38,231 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:38,231 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. after waiting 0 ms 2023-02-12 03:58:38,231 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:38,231 INFO [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:38,231 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-0] regionserver.HRegion(1557): Region close journal for cc0e0b01138d4eae5df1e5641b484f45: Waiting for close lock at 1676174318229Disabling compacts and flushes for region at 1676174318229Disabling writes for close at 1676174318231 (+2 ms)Writing region close event to WAL at 1676174318231Closed at 1676174318231 2023-02-12 03:58:38,232 INFO [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-0] regionserver.HRegion(7671): creating {ENCODED => b6894e9d4c505cdcd16894575e6b138a, NAME => 'testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676174318078', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp 2023-02-12 03:58:38,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-12 03:58:38,684 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:38,685 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:38,685 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:38,685 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-0] regionserver.HRegion(1603): Closing b6894e9d4c505cdcd16894575e6b138a, disabling compactions & flushes 2023-02-12 03:58:38,685 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-1] regionserver.HRegion(1603): Closing e019e3f4c00b3da154d59b4085911d76, disabling compactions & flushes 2023-02-12 03:58:38,685 INFO [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:38,685 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-2] regionserver.HRegion(1603): Closing 609b1a331ab507675f39b4bdbe7a382c, disabling compactions & flushes 2023-02-12 03:58:38,685 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:38,685 INFO [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:38,685 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. after waiting 0 ms 2023-02-12 03:58:38,685 INFO [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:38,685 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:38,685 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:38,686 INFO [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:38,685 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:38,686 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-0] regionserver.HRegion(1557): Region close journal for b6894e9d4c505cdcd16894575e6b138a: Waiting for close lock at 1676174318685Disabling compacts and flushes for region at 1676174318685Disabling writes for close at 1676174318685Writing region close event to WAL at 1676174318686 (+1 ms)Closed at 1676174318686 2023-02-12 03:58:38,686 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. after waiting 0 ms 2023-02-12 03:58:38,686 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. after waiting 0 ms 2023-02-12 03:58:38,686 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:38,686 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:38,686 INFO [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:38,686 INFO [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:38,686 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-1] regionserver.HRegion(1557): Region close journal for e019e3f4c00b3da154d59b4085911d76: Waiting for close lock at 1676174318685Disabling compacts and flushes for region at 1676174318685Disabling writes for close at 1676174318686 (+1 ms)Writing region close event to WAL at 1676174318686Closed at 1676174318686 2023-02-12 03:58:38,686 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676174318078-pool-2] regionserver.HRegion(1557): Region close journal for 609b1a331ab507675f39b4bdbe7a382c: Waiting for close lock at 1676174318685Disabling compacts and flushes for region at 1676174318685Disabling writes for close at 1676174318686 (+1 ms)Writing region close event to WAL at 1676174318686Closed at 1676174318686 2023-02-12 03:58:38,691 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676174318078 execute state=CREATE_TABLE_ADD_TO_META 2023-02-12 03:58:38,693 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174318692"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174318692"}]},"ts":"1676174318692"} 2023-02-12 03:58:38,693 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174318692"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174318692"}]},"ts":"1676174318692"} 2023-02-12 03:58:38,693 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174318692"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174318692"}]},"ts":"1676174318692"} 2023-02-12 03:58:38,694 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174318692"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174318692"}]},"ts":"1676174318692"} 2023-02-12 03:58:38,694 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174318692"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174318692"}]},"ts":"1676174318692"} 2023-02-12 03:58:38,694 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174318692"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174318692"}]},"ts":"1676174318692"} 2023-02-12 03:58:38,767 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-12 03:58:38,769 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676174318078 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-12 03:58:38,770 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174318769"}]},"ts":"1676174318769"} 2023-02-12 03:58:38,772 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=ENABLING in hbase:meta 2023-02-12 03:58:38,776 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-02-12 03:58:38,782 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-12 03:58:38,782 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-12 03:58:38,782 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-12 03:58:38,782 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-12 03:58:38,783 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN}, {pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN}, {pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN}] 2023-02-12 03:58:38,787 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN 2023-02-12 03:58:38,787 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN 2023-02-12 03:58:38,788 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN 2023-02-12 03:58:38,788 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN 2023-02-12 03:58:38,790 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,42205,1676174315413; forceNewPlan=false, retain=false 2023-02-12 03:58:38,790 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,42205,1676174315413; forceNewPlan=false, retain=false 2023-02-12 03:58:38,790 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN 2023-02-12 03:58:38,790 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,35783,1676174315451; forceNewPlan=false, retain=false 2023-02-12 03:58:38,790 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,35785,1676174315481; forceNewPlan=false, retain=false 2023-02-12 03:58:38,792 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN 2023-02-12 03:58:38,792 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,35783,1676174315451; forceNewPlan=false, retain=false 2023-02-12 03:58:38,793 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,35785,1676174315481; forceNewPlan=false, retain=false 2023-02-12 03:58:38,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-12 03:58:38,940 INFO [jenkins-hbase20:37305] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-12 03:58:38,943 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:38,943 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:38,943 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:38,943 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:38,943 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:38,943 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174318943"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174318943"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174318943"}]},"ts":"1676174318943"} 2023-02-12 03:58:38,943 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174318943"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174318943"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174318943"}]},"ts":"1676174318943"} 2023-02-12 03:58:38,944 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174318943"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174318943"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174318943"}]},"ts":"1676174318943"} 2023-02-12 03:58:38,943 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174318943"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174318943"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174318943"}]},"ts":"1676174318943"} 2023-02-12 03:58:38,943 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174318943"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174318943"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174318943"}]},"ts":"1676174318943"} 2023-02-12 03:58:38,946 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=13, state=RUNNABLE; OpenRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:38,953 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=11, state=RUNNABLE; OpenRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:38,956 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:38,956 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174318956"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174318956"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174318956"}]},"ts":"1676174318956"} 2023-02-12 03:58:38,957 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=14, state=RUNNABLE; OpenRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:38,966 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=12, state=RUNNABLE; OpenRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:38,969 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=15, state=RUNNABLE; OpenRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:38,973 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=10, state=RUNNABLE; OpenRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:39,108 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:39,108 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 609b1a331ab507675f39b4bdbe7a382c, NAME => 'testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.', STARTKEY => '3', ENDKEY => '4'} 2023-02-12 03:58:39,108 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:39,109 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:39,109 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:39,109 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:39,110 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:39,110 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-12 03:58:39,112 INFO [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:39,117 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52216, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-12 03:58:39,118 DEBUG [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf 2023-02-12 03:58:39,118 DEBUG [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf 2023-02-12 03:58:39,120 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:39,121 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-12 03:58:39,121 INFO [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 609b1a331ab507675f39b4bdbe7a382c columnFamilyName cf 2023-02-12 03:58:39,122 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:39,123 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ecda59e01589f4f279b19a5420a1e608, NAME => 'testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.', STARTKEY => '1', ENDKEY => '2'} 2023-02-12 03:58:39,123 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44136, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-12 03:58:39,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:39,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:39,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:39,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:39,126 INFO [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] regionserver.HStore(310): Store=609b1a331ab507675f39b4bdbe7a382c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:39,127 INFO [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:39,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:39,130 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:39,131 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:39,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => fcd2ed887eb9f47c07236428879982c1, NAME => 'testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.', STARTKEY => '2', ENDKEY => '3'} 2023-02-12 03:58:39,131 DEBUG [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf 2023-02-12 03:58:39,131 DEBUG [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf 2023-02-12 03:58:39,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:39,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:39,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:39,131 INFO [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ecda59e01589f4f279b19a5420a1e608 columnFamilyName cf 2023-02-12 03:58:39,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:39,133 INFO [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] regionserver.HStore(310): Store=ecda59e01589f4f279b19a5420a1e608/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:39,139 INFO [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:39,139 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:39,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:39,141 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:39,142 DEBUG [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf 2023-02-12 03:58:39,142 DEBUG [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf 2023-02-12 03:58:39,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 03:58:39,143 INFO [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fcd2ed887eb9f47c07236428879982c1 columnFamilyName cf 2023-02-12 03:58:39,143 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 609b1a331ab507675f39b4bdbe7a382c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62737178, jitterRate=-0.06514319777488708}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:39,143 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 609b1a331ab507675f39b4bdbe7a382c: Running coprocessor pre-open hook at 1676174319109Writing region info on filesystem at 1676174319109Initializing all the Stores at 1676174319110 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174319110Cleaning up temporary data from old regions at 1676174319131 (+21 ms)Cleaning up detritus from prior splits at 1676174319132 (+1 ms)Running coprocessor post-open hooks at 1676174319143 (+11 ms)Region opened successfully at 1676174319143 2023-02-12 03:58:39,145 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c., pid=16, masterSystemTime=1676174319102 2023-02-12 03:58:39,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:39,148 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:39,148 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:39,149 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => cc0e0b01138d4eae5df1e5641b484f45, NAME => 'testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.', STARTKEY => '', ENDKEY => '1'} 2023-02-12 03:58:39,149 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:39,155 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174319149"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174319149"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174319149"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174319149"}]},"ts":"1676174319149"} 2023-02-12 03:58:39,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:39,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:39,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:39,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:39,156 INFO [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] regionserver.HStore(310): Store=fcd2ed887eb9f47c07236428879982c1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:39,158 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:39,158 INFO [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:39,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 03:58:39,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:39,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:39,166 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened ecda59e01589f4f279b19a5420a1e608; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65975300, jitterRate=-0.016891419887542725}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:39,167 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for ecda59e01589f4f279b19a5420a1e608: Running coprocessor pre-open hook at 1676174319125Writing region info on filesystem at 1676174319125Initializing all the Stores at 1676174319126 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174319126Cleaning up temporary data from old regions at 1676174319141 (+15 ms)Cleaning up detritus from prior splits at 1676174319142 (+1 ms)Running coprocessor post-open hooks at 1676174319167 (+25 ms)Region opened successfully at 1676174319167 2023-02-12 03:58:39,167 DEBUG [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf 2023-02-12 03:58:39,168 DEBUG [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf 2023-02-12 03:58:39,168 INFO [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cc0e0b01138d4eae5df1e5641b484f45 columnFamilyName cf 2023-02-12 03:58:39,169 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=13 2023-02-12 03:58:39,170 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=13, state=SUCCESS; OpenRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451 in 213 msec 2023-02-12 03:58:39,170 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608., pid=17, masterSystemTime=1676174319110 2023-02-12 03:58:39,174 INFO [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] regionserver.HStore(310): Store=cc0e0b01138d4eae5df1e5641b484f45/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:39,178 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN in 387 msec 2023-02-12 03:58:39,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:39,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:39,186 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:39,187 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:39,187 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174319186"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174319186"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174319186"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174319186"}]},"ts":"1676174319186"} 2023-02-12 03:58:39,188 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:39,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:39,192 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:39,196 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=11 2023-02-12 03:58:39,197 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=11, state=SUCCESS; OpenRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,42205,1676174315413 in 239 msec 2023-02-12 03:58:39,189 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:39,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e019e3f4c00b3da154d59b4085911d76, NAME => 'testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.', STARTKEY => '4', ENDKEY => '5'} 2023-02-12 03:58:39,201 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN in 414 msec 2023-02-12 03:58:39,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:39,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:39,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:39,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:39,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 03:58:39,220 INFO [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:39,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 03:58:39,221 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened cc0e0b01138d4eae5df1e5641b484f45; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61717736, jitterRate=-0.08033406734466553}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:39,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for cc0e0b01138d4eae5df1e5641b484f45: Running coprocessor pre-open hook at 1676174319155Writing region info on filesystem at 1676174319155Initializing all the Stores at 1676174319158 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174319158Cleaning up temporary data from old regions at 1676174319187 (+29 ms)Cleaning up detritus from prior splits at 1676174319189 (+2 ms)Running coprocessor post-open hooks at 1676174319221 (+32 ms)Region opened successfully at 1676174319221 2023-02-12 03:58:39,221 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened fcd2ed887eb9f47c07236428879982c1; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63786997, jitterRate=-0.04949967563152313}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:39,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for fcd2ed887eb9f47c07236428879982c1: Running coprocessor pre-open hook at 1676174319132Writing region info on filesystem at 1676174319132Initializing all the Stores at 1676174319133 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174319134 (+1 ms)Cleaning up temporary data from old regions at 1676174319168 (+34 ms)Cleaning up detritus from prior splits at 1676174319169 (+1 ms)Running coprocessor post-open hooks at 1676174319221 (+52 ms)Region opened successfully at 1676174319221 2023-02-12 03:58:39,224 DEBUG [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf 2023-02-12 03:58:39,225 DEBUG [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf 2023-02-12 03:58:39,225 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45., pid=21, masterSystemTime=1676174319102 2023-02-12 03:58:39,226 INFO [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e019e3f4c00b3da154d59b4085911d76 columnFamilyName cf 2023-02-12 03:58:39,227 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1., pid=19, masterSystemTime=1676174319120 2023-02-12 03:58:39,229 INFO [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] regionserver.HStore(310): Store=e019e3f4c00b3da154d59b4085911d76/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:39,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:39,231 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:39,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:39,232 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:39,233 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174319232"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174319232"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174319232"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174319232"}]},"ts":"1676174319232"} 2023-02-12 03:58:39,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:39,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:39,234 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:39,234 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:39,234 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b6894e9d4c505cdcd16894575e6b138a, NAME => 'testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.', STARTKEY => '5', ENDKEY => ''} 2023-02-12 03:58:39,234 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:39,235 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:39,235 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:39,235 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:39,236 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:39,236 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174319236"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174319236"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174319236"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174319236"}]},"ts":"1676174319236"} 2023-02-12 03:58:39,239 INFO [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:39,242 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:39,243 DEBUG [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf 2023-02-12 03:58:39,244 DEBUG [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf 2023-02-12 03:58:39,245 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=10 2023-02-12 03:58:39,245 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=10, state=SUCCESS; OpenRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35783,1676174315451 in 264 msec 2023-02-12 03:58:39,245 INFO [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b6894e9d4c505cdcd16894575e6b138a columnFamilyName cf 2023-02-12 03:58:39,247 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=12 2023-02-12 03:58:39,247 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=12, state=SUCCESS; OpenRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,35785,1676174315481 in 274 msec 2023-02-12 03:58:39,247 INFO [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] regionserver.HStore(310): Store=b6894e9d4c505cdcd16894575e6b138a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:39,255 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 03:58:39,255 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:39,255 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened e019e3f4c00b3da154d59b4085911d76; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70368815, jitterRate=0.04857705533504486}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:39,256 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN in 462 msec 2023-02-12 03:58:39,256 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for e019e3f4c00b3da154d59b4085911d76: Running coprocessor pre-open hook at 1676174319202Writing region info on filesystem at 1676174319202Initializing all the Stores at 1676174319203 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174319203Cleaning up temporary data from old regions at 1676174319235 (+32 ms)Cleaning up detritus from prior splits at 1676174319235Running coprocessor post-open hooks at 1676174319256 (+21 ms)Region opened successfully at 1676174319256 2023-02-12 03:58:39,256 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:39,259 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76., pid=18, masterSystemTime=1676174319110 2023-02-12 03:58:39,260 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN in 464 msec 2023-02-12 03:58:39,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:39,263 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:39,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:39,264 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:39,264 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174319264"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174319264"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174319264"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174319264"}]},"ts":"1676174319264"} 2023-02-12 03:58:39,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 03:58:39,268 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened b6894e9d4c505cdcd16894575e6b138a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69033953, jitterRate=0.02868606150150299}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:39,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for b6894e9d4c505cdcd16894575e6b138a: Running coprocessor pre-open hook at 1676174319235Writing region info on filesystem at 1676174319235Initializing all the Stores at 1676174319239 (+4 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174319239Cleaning up temporary data from old regions at 1676174319257 (+18 ms)Cleaning up detritus from prior splits at 1676174319259 (+2 ms)Running coprocessor post-open hooks at 1676174319268 (+9 ms)Region opened successfully at 1676174319268 2023-02-12 03:58:39,272 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=14 2023-02-12 03:58:39,272 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=14, state=SUCCESS; OpenRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,42205,1676174315413 in 310 msec 2023-02-12 03:58:39,275 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN in 489 msec 2023-02-12 03:58:39,276 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a., pid=20, masterSystemTime=1676174319120 2023-02-12 03:58:39,279 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:39,279 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:39,280 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:39,280 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174319280"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174319280"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174319280"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174319280"}]},"ts":"1676174319280"} 2023-02-12 03:58:39,286 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=15 2023-02-12 03:58:39,286 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=15, state=SUCCESS; OpenRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,35785,1676174315481 in 314 msec 2023-02-12 03:58:39,290 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=9 2023-02-12 03:58:39,290 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN in 503 msec 2023-02-12 03:58:39,291 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676174318078 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-12 03:58:39,292 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174319291"}]},"ts":"1676174319291"} 2023-02-12 03:58:39,294 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=ENABLED in hbase:meta 2023-02-12 03:58:39,297 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676174318078 execute state=CREATE_TABLE_POST_OPERATION 2023-02-12 03:58:39,301 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testRestoreSnapshot-1676174318078 in 1.1950 sec 2023-02-12 03:58:39,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-12 03:58:39,632 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRestoreSnapshot-1676174318078, procId: 9 completed 2023-02-12 03:58:39,638 DEBUG [Listener at localhost.localdomain/46483] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676174318078 2023-02-12 03:58:39,639 DEBUG [Listener at localhost.localdomain/46483] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:39,642 INFO [Listener at localhost.localdomain/46483] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-12 03:58:39,671 DEBUG [hconnection-0x51e75f37-shared-pool-5] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:39,678 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52226, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:39,679 DEBUG [hconnection-0x51e75f37-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:39,682 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44142, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:39,694 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676174318078 2023-02-12 03:58:39,698 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$11(2405): Client=jenkins//148.251.75.209 disable testRestoreSnapshot-1676174318078 2023-02-12 03:58:39,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=22, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:39,710 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174319710"}]},"ts":"1676174319710"} 2023-02-12 03:58:39,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-12 03:58:39,712 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=DISABLING in hbase:meta 2023-02-12 03:58:39,715 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676174318078 to state=DISABLING 2023-02-12 03:58:39,724 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, UNASSIGN}, {pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, UNASSIGN}, {pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, UNASSIGN}, {pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, UNASSIGN}, {pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, UNASSIGN}, {pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, UNASSIGN}] 2023-02-12 03:58:39,727 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, UNASSIGN 2023-02-12 03:58:39,727 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, UNASSIGN 2023-02-12 03:58:39,727 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, UNASSIGN 2023-02-12 03:58:39,728 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, UNASSIGN 2023-02-12 03:58:39,728 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, UNASSIGN 2023-02-12 03:58:39,731 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:39,731 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:39,731 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:39,731 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:39,731 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:39,732 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174319731"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174319731"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174319731"}]},"ts":"1676174319731"} 2023-02-12 03:58:39,732 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174319731"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174319731"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174319731"}]},"ts":"1676174319731"} 2023-02-12 03:58:39,732 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174319731"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174319731"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174319731"}]},"ts":"1676174319731"} 2023-02-12 03:58:39,732 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174319731"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174319731"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174319731"}]},"ts":"1676174319731"} 2023-02-12 03:58:39,733 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174319731"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174319731"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174319731"}]},"ts":"1676174319731"} 2023-02-12 03:58:39,734 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=23, state=RUNNABLE; CloseRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:39,739 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=25, state=RUNNABLE; CloseRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:39,742 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=28, state=RUNNABLE; CloseRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:39,744 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=24, state=RUNNABLE; CloseRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:39,746 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=27, state=RUNNABLE; CloseRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:39,756 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, UNASSIGN 2023-02-12 03:58:39,760 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:39,761 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174319760"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174319760"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174319760"}]},"ts":"1676174319760"} 2023-02-12 03:58:39,764 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=26, state=RUNNABLE; CloseRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:39,898 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:39,899 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing cc0e0b01138d4eae5df1e5641b484f45, disabling compactions & flushes 2023-02-12 03:58:39,899 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:39,899 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:39,899 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. after waiting 0 ms 2023-02-12 03:58:39,899 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:39,907 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:39,908 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing b6894e9d4c505cdcd16894575e6b138a, disabling compactions & flushes 2023-02-12 03:58:39,908 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:39,908 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:39,908 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. after waiting 0 ms 2023-02-12 03:58:39,909 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:39,911 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 03:58:39,914 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:39,915 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing ecda59e01589f4f279b19a5420a1e608, disabling compactions & flushes 2023-02-12 03:58:39,915 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:39,915 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:39,915 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. after waiting 0 ms 2023-02-12 03:58:39,915 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:39,915 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:39,915 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for cc0e0b01138d4eae5df1e5641b484f45: Waiting for close lock at 1676174319899Running coprocessor pre-close hooks at 1676174319899Disabling compacts and flushes for region at 1676174319899Disabling writes for close at 1676174319899Writing region close event to WAL at 1676174319899Running coprocessor post-close hooks at 1676174319915 (+16 ms)Closed at 1676174319915 2023-02-12 03:58:39,922 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:39,922 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:39,922 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 609b1a331ab507675f39b4bdbe7a382c, disabling compactions & flushes 2023-02-12 03:58:39,922 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:39,922 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:39,922 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. after waiting 0 ms 2023-02-12 03:58:39,922 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:39,925 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=CLOSED 2023-02-12 03:58:39,926 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174319925"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174319925"}]},"ts":"1676174319925"} 2023-02-12 03:58:39,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 03:58:39,932 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:39,932 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for ecda59e01589f4f279b19a5420a1e608: Waiting for close lock at 1676174319914Running coprocessor pre-close hooks at 1676174319914Disabling compacts and flushes for region at 1676174319914Disabling writes for close at 1676174319915 (+1 ms)Writing region close event to WAL at 1676174319917 (+2 ms)Running coprocessor post-close hooks at 1676174319932 (+15 ms)Closed at 1676174319932 2023-02-12 03:58:39,933 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 03:58:39,935 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:39,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 609b1a331ab507675f39b4bdbe7a382c: Waiting for close lock at 1676174319922Running coprocessor pre-close hooks at 1676174319922Disabling compacts and flushes for region at 1676174319922Disabling writes for close at 1676174319922Writing region close event to WAL at 1676174319923 (+1 ms)Running coprocessor post-close hooks at 1676174319935 (+12 ms)Closed at 1676174319935 2023-02-12 03:58:39,937 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:39,937 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:39,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing e019e3f4c00b3da154d59b4085911d76, disabling compactions & flushes 2023-02-12 03:58:39,938 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:39,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:39,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. after waiting 0 ms 2023-02-12 03:58:39,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:39,939 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 03:58:39,940 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:39,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for b6894e9d4c505cdcd16894575e6b138a: Waiting for close lock at 1676174319907Running coprocessor pre-close hooks at 1676174319907Disabling compacts and flushes for region at 1676174319907Disabling writes for close at 1676174319909 (+2 ms)Writing region close event to WAL at 1676174319913 (+4 ms)Running coprocessor post-close hooks at 1676174319940 (+27 ms)Closed at 1676174319940 2023-02-12 03:58:39,944 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=CLOSED 2023-02-12 03:58:39,944 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174319943"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174319943"}]},"ts":"1676174319943"} 2023-02-12 03:58:39,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:39,946 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=CLOSED 2023-02-12 03:58:39,946 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174319946"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174319946"}]},"ts":"1676174319946"} 2023-02-12 03:58:39,948 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=23 2023-02-12 03:58:39,948 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=23, state=SUCCESS; CloseRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35783,1676174315451 in 201 msec 2023-02-12 03:58:39,950 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:39,950 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:39,951 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing fcd2ed887eb9f47c07236428879982c1, disabling compactions & flushes 2023-02-12 03:58:39,951 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:39,951 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:39,951 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. after waiting 0 ms 2023-02-12 03:58:39,951 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:39,956 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=CLOSED 2023-02-12 03:58:39,956 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174319956"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174319956"}]},"ts":"1676174319956"} 2023-02-12 03:58:39,958 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, UNASSIGN in 225 msec 2023-02-12 03:58:39,960 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=24 2023-02-12 03:58:39,960 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=24, state=SUCCESS; CloseRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,42205,1676174315413 in 204 msec 2023-02-12 03:58:39,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 03:58:39,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-12 03:58:39,964 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=26 2023-02-12 03:58:39,964 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:39,964 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=26, state=SUCCESS; CloseRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451 in 191 msec 2023-02-12 03:58:39,964 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for fcd2ed887eb9f47c07236428879982c1: Waiting for close lock at 1676174319950Running coprocessor pre-close hooks at 1676174319950Disabling compacts and flushes for region at 1676174319950Disabling writes for close at 1676174319951 (+1 ms)Writing region close event to WAL at 1676174319955 (+4 ms)Running coprocessor post-close hooks at 1676174319964 (+9 ms)Closed at 1676174319964 2023-02-12 03:58:39,967 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, UNASSIGN in 237 msec 2023-02-12 03:58:39,969 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, UNASSIGN in 240 msec 2023-02-12 03:58:39,969 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=28 2023-02-12 03:58:39,969 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:39,970 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=28, state=SUCCESS; CloseRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,35785,1676174315481 in 219 msec 2023-02-12 03:58:39,971 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=CLOSED 2023-02-12 03:58:39,972 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174319971"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174319971"}]},"ts":"1676174319971"} 2023-02-12 03:58:39,981 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 03:58:39,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:39,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for e019e3f4c00b3da154d59b4085911d76: Waiting for close lock at 1676174319937Running coprocessor pre-close hooks at 1676174319937Disabling compacts and flushes for region at 1676174319937Disabling writes for close at 1676174319938 (+1 ms)Writing region close event to WAL at 1676174319961 (+23 ms)Running coprocessor post-close hooks at 1676174319982 (+21 ms)Closed at 1676174319982 2023-02-12 03:58:39,983 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, UNASSIGN in 246 msec 2023-02-12 03:58:39,993 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:39,993 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=CLOSED 2023-02-12 03:58:39,994 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174319993"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174319993"}]},"ts":"1676174319993"} 2023-02-12 03:58:39,994 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=25 2023-02-12 03:58:39,995 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=25, state=SUCCESS; CloseRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,35785,1676174315481 in 244 msec 2023-02-12 03:58:39,996 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, UNASSIGN in 271 msec 2023-02-12 03:58:39,998 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=27 2023-02-12 03:58:39,999 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=27, state=SUCCESS; CloseRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,42205,1676174315413 in 250 msec 2023-02-12 03:58:40,001 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=22 2023-02-12 03:58:40,001 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, UNASSIGN in 275 msec 2023-02-12 03:58:40,002 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174320002"}]},"ts":"1676174320002"} 2023-02-12 03:58:40,004 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=DISABLED in hbase:meta 2023-02-12 03:58:40,006 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676174318078 to state=DISABLED 2023-02-12 03:58:40,011 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=22, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676174318078 in 307 msec 2023-02-12 03:58:40,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-12 03:58:40,465 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676174318078, procId: 22 completed 2023-02-12 03:58:40,476 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1580): Client=jenkins//148.251.75.209 snapshot request for:{ ss=emptySnaptb-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } 2023-02-12 03:58:40,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676174320476 (current time:1676174320476). 2023-02-12 03:58:40,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-12 03:58:40,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] zookeeper.ReadOnlyZKClient(139): Connect 0x704646a2 to 127.0.0.1:50368 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 03:58:40,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bef7bd2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 03:58:40,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:40,498 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47314, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:40,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x704646a2 to 127.0.0.1:50368 2023-02-12 03:58:40,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:58:40,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-12 03:58:40,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } 2023-02-12 03:58:40,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE 2023-02-12 03:58:40,530 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE 2023-02-12 03:58:40,532 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=35, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE; timeout=600000, timestamp=1676174920531 2023-02-12 03:58:40,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } 2023-02-12 03:58:40,535 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676174318078 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676174318078 2023-02-12 03:58:40,538 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-12 03:58:40,538 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-12 03:58:40,582 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676174318078 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } 2023-02-12 03:58:40,594 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:40,594 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:40,595 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:40,596 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:40,598 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:40,598 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:40,598 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:40,601 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:40,601 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:40,601 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:40,601 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:40,601 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:40,601 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:40,601 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:40,611 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 03:58:40,611 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 03:58:40,612 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 03:58:40,613 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 03:58:40,614 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 03:58:40,617 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 03:58:40,617 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 03:58:40,725 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676174318078 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } as finished. 2023-02-12 03:58:40,725 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676174318078 2023-02-12 03:58:40,728 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/emptySnaptb-1676174318078 2023-02-12 03:58:40,769 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/emptySnaptb-1676174318078 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/emptySnaptb-1676174318078 2023-02-12 03:58:40,789 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 03:58:40,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } is done 2023-02-12 03:58:40,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(404): Snapshoting '{ ss=emptySnaptb-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 }' is still in progress! 2023-02-12 03:58:40,799 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-12 03:58:40,814 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676174318078 of table testRestoreSnapshot-1676174318078 completed 2023-02-12 03:58:40,814 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/emptySnaptb-1676174318078 2023-02-12 03:58:40,815 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/emptySnaptb-1676174318078 2023-02-12 03:58:40,815 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676174318078 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676174318078 at 1676174320535Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676174318078 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } at 1676174320582 (+47 ms)Consolidate snapshot: emptySnaptb-1676174318078 at 1676174320725 (+143 ms)Loading Region manifests for emptySnaptb-1676174318078 at 1676174320726 (+1 ms)Writing data manifest for emptySnaptb-1676174318078 at 1676174320738 (+12 ms)Verifying snapshot: emptySnaptb-1676174318078 at 1676174320755 (+17 ms)Snapshot emptySnaptb-1676174318078 of table testRestoreSnapshot-1676174318078 completed at 1676174320814 (+59 ms) 2023-02-12 03:58:40,816 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=35, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE 2023-02-12 03:58:40,819 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE in 302 msec 2023-02-12 03:58:41,300 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 03:58:41,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } is done 2023-02-12 03:58:41,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-12 03:58:41,303 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676174318078 2023-02-12 03:58:41,306 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$10(2372): Client=jenkins//148.251.75.209 enable testRestoreSnapshot-1676174318078 2023-02-12 03:58:41,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=36, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:41,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-12 03:58:41,316 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676174318078 2023-02-12 03:58:41,316 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174321316"}]},"ts":"1676174321316"} 2023-02-12 03:58:41,318 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=ENABLING in hbase:meta 2023-02-12 03:58:41,321 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-12 03:58:41,324 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN}, {pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN}, {pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN}, {pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN}, {pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN}, {pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN}] 2023-02-12 03:58:41,326 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN 2023-02-12 03:58:41,327 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN 2023-02-12 03:58:41,327 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN 2023-02-12 03:58:41,327 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN 2023-02-12 03:58:41,329 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:41,329 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:41,330 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN 2023-02-12 03:58:41,330 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,35783,1676174315451; forceNewPlan=false, retain=true 2023-02-12 03:58:41,329 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:41,329 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:41,330 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,35783,1676174315451; forceNewPlan=false, retain=true 2023-02-12 03:58:41,330 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,42205,1676174315413; forceNewPlan=false, retain=true 2023-02-12 03:58:41,330 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,35785,1676174315481; forceNewPlan=false, retain=true 2023-02-12 03:58:41,331 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:41,331 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN 2023-02-12 03:58:41,331 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,35785,1676174315481; forceNewPlan=false, retain=true 2023-02-12 03:58:41,331 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:41,332 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,42205,1676174315413; forceNewPlan=false, retain=true 2023-02-12 03:58:41,480 INFO [jenkins-hbase20:37305] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-12 03:58:41,480 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:41,480 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:41,481 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174321480"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174321480"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174321480"}]},"ts":"1676174321480"} 2023-02-12 03:58:41,480 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:41,480 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:41,480 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:41,481 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174321480"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174321480"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174321480"}]},"ts":"1676174321480"} 2023-02-12 03:58:41,481 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174321480"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174321480"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174321480"}]},"ts":"1676174321480"} 2023-02-12 03:58:41,481 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174321480"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174321480"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174321480"}]},"ts":"1676174321480"} 2023-02-12 03:58:41,481 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174321480"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174321480"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174321480"}]},"ts":"1676174321480"} 2023-02-12 03:58:41,485 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=42, state=RUNNABLE; OpenRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:41,487 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=39, state=RUNNABLE; OpenRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:41,489 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=40, state=RUNNABLE; OpenRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:41,491 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=38, state=RUNNABLE; OpenRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:41,491 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:41,491 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174321491"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174321491"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174321491"}]},"ts":"1676174321491"} 2023-02-12 03:58:41,492 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=47, ppid=41, state=RUNNABLE; OpenRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:41,493 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=37, state=RUNNABLE; OpenRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:41,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-12 03:58:41,648 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:41,649 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => fcd2ed887eb9f47c07236428879982c1, NAME => 'testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.', STARTKEY => '2', ENDKEY => '3'} 2023-02-12 03:58:41,649 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:41,649 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:41,649 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:41,649 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:41,650 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:41,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => cc0e0b01138d4eae5df1e5641b484f45, NAME => 'testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.', STARTKEY => '', ENDKEY => '1'} 2023-02-12 03:58:41,650 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:41,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ecda59e01589f4f279b19a5420a1e608, NAME => 'testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.', STARTKEY => '1', ENDKEY => '2'} 2023-02-12 03:58:41,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:41,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:41,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:41,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:41,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:41,651 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:41,651 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:41,651 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:41,651 INFO [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:41,652 INFO [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:41,652 INFO [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:41,653 DEBUG [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf 2023-02-12 03:58:41,653 DEBUG [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf 2023-02-12 03:58:41,653 INFO [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fcd2ed887eb9f47c07236428879982c1 columnFamilyName cf 2023-02-12 03:58:41,653 DEBUG [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf 2023-02-12 03:58:41,653 DEBUG [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf 2023-02-12 03:58:41,654 DEBUG [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf 2023-02-12 03:58:41,654 DEBUG [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf 2023-02-12 03:58:41,654 INFO [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] regionserver.HStore(310): Store=fcd2ed887eb9f47c07236428879982c1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:41,655 INFO [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ecda59e01589f4f279b19a5420a1e608 columnFamilyName cf 2023-02-12 03:58:41,655 INFO [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cc0e0b01138d4eae5df1e5641b484f45 columnFamilyName cf 2023-02-12 03:58:41,656 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:41,656 INFO [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] regionserver.HStore(310): Store=ecda59e01589f4f279b19a5420a1e608/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:41,656 INFO [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] regionserver.HStore(310): Store=cc0e0b01138d4eae5df1e5641b484f45/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:41,657 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:41,657 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:41,657 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:41,659 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:41,659 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:41,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:41,661 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened fcd2ed887eb9f47c07236428879982c1; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70783125, jitterRate=0.05475075542926788}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:41,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for fcd2ed887eb9f47c07236428879982c1: Running coprocessor pre-open hook at 1676174321650Writing region info on filesystem at 1676174321650Initializing all the Stores at 1676174321651 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174321651Cleaning up temporary data from old regions at 1676174321657 (+6 ms)Cleaning up detritus from prior splits at 1676174321658 (+1 ms)Running coprocessor post-open hooks at 1676174321661 (+3 ms)Region opened successfully at 1676174321661 2023-02-12 03:58:41,662 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1., pid=44, masterSystemTime=1676174321639 2023-02-12 03:58:41,662 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:41,662 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:41,664 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened ecda59e01589f4f279b19a5420a1e608; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71975092, jitterRate=0.07251244783401489}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:41,664 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened cc0e0b01138d4eae5df1e5641b484f45; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70927102, jitterRate=0.05689617991447449}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:41,664 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for ecda59e01589f4f279b19a5420a1e608: Running coprocessor pre-open hook at 1676174321651Writing region info on filesystem at 1676174321651Initializing all the Stores at 1676174321652 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174321652Cleaning up temporary data from old regions at 1676174321659 (+7 ms)Cleaning up detritus from prior splits at 1676174321659Running coprocessor post-open hooks at 1676174321664 (+5 ms)Region opened successfully at 1676174321664 2023-02-12 03:58:41,664 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for cc0e0b01138d4eae5df1e5641b484f45: Running coprocessor pre-open hook at 1676174321651Writing region info on filesystem at 1676174321651Initializing all the Stores at 1676174321652 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174321652Cleaning up temporary data from old regions at 1676174321659 (+7 ms)Cleaning up detritus from prior splits at 1676174321660 (+1 ms)Running coprocessor post-open hooks at 1676174321664 (+4 ms)Region opened successfully at 1676174321664 2023-02-12 03:58:41,664 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:41,665 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:41,665 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:41,665 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45., pid=48, masterSystemTime=1676174321642 2023-02-12 03:58:41,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b6894e9d4c505cdcd16894575e6b138a, NAME => 'testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.', STARTKEY => '5', ENDKEY => ''} 2023-02-12 03:58:41,665 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:41,665 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174321665"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174321665"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174321665"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174321665"}]},"ts":"1676174321665"} 2023-02-12 03:58:41,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:41,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:41,665 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608., pid=46, masterSystemTime=1676174321643 2023-02-12 03:58:41,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:41,666 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:41,668 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:41,668 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:41,668 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:41,668 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 609b1a331ab507675f39b4bdbe7a382c, NAME => 'testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.', STARTKEY => '3', ENDKEY => '4'} 2023-02-12 03:58:41,668 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:41,669 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:41,669 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:41,669 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:41,669 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:41,670 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174321669"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174321669"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174321669"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174321669"}]},"ts":"1676174321669"} 2023-02-12 03:58:41,670 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:41,670 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:41,670 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:41,671 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e019e3f4c00b3da154d59b4085911d76, NAME => 'testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.', STARTKEY => '4', ENDKEY => '5'} 2023-02-12 03:58:41,671 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:41,671 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:41,671 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:41,671 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:41,671 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:41,671 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174321671"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174321671"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174321671"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174321671"}]},"ts":"1676174321671"} 2023-02-12 03:58:41,674 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=39 2023-02-12 03:58:41,674 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=39, state=SUCCESS; OpenRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,35785,1676174315481 in 180 msec 2023-02-12 03:58:41,675 INFO [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:41,675 INFO [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:41,676 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN in 353 msec 2023-02-12 03:58:41,677 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=37 2023-02-12 03:58:41,677 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=37, state=SUCCESS; OpenRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35783,1676174315451 in 181 msec 2023-02-12 03:58:41,677 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=38 2023-02-12 03:58:41,678 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=38, state=SUCCESS; OpenRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,42205,1676174315413 in 185 msec 2023-02-12 03:58:41,678 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN in 356 msec 2023-02-12 03:58:41,679 INFO [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:41,680 DEBUG [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf 2023-02-12 03:58:41,680 DEBUG [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf 2023-02-12 03:58:41,680 DEBUG [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf 2023-02-12 03:58:41,680 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN in 357 msec 2023-02-12 03:58:41,680 DEBUG [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf 2023-02-12 03:58:41,680 INFO [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e019e3f4c00b3da154d59b4085911d76 columnFamilyName cf 2023-02-12 03:58:41,680 INFO [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b6894e9d4c505cdcd16894575e6b138a columnFamilyName cf 2023-02-12 03:58:41,680 DEBUG [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf 2023-02-12 03:58:41,680 DEBUG [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf 2023-02-12 03:58:41,681 INFO [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 609b1a331ab507675f39b4bdbe7a382c columnFamilyName cf 2023-02-12 03:58:41,681 INFO [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] regionserver.HStore(310): Store=b6894e9d4c505cdcd16894575e6b138a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:41,682 INFO [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] regionserver.HStore(310): Store=e019e3f4c00b3da154d59b4085911d76/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:41,682 INFO [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] regionserver.HStore(310): Store=609b1a331ab507675f39b4bdbe7a382c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:41,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:41,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:41,684 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:41,684 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:41,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:41,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:41,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:41,689 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 609b1a331ab507675f39b4bdbe7a382c; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62205529, jitterRate=-0.073065385222435}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:41,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 609b1a331ab507675f39b4bdbe7a382c: Running coprocessor pre-open hook at 1676174321669Writing region info on filesystem at 1676174321669Initializing all the Stores at 1676174321671 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174321671Cleaning up temporary data from old regions at 1676174321685 (+14 ms)Cleaning up detritus from prior splits at 1676174321685Running coprocessor post-open hooks at 1676174321689 (+4 ms)Region opened successfully at 1676174321689 2023-02-12 03:58:41,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:41,689 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened e019e3f4c00b3da154d59b4085911d76; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65244215, jitterRate=-0.02778543531894684}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:41,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for e019e3f4c00b3da154d59b4085911d76: Running coprocessor pre-open hook at 1676174321671Writing region info on filesystem at 1676174321671Initializing all the Stores at 1676174321673 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174321673Cleaning up temporary data from old regions at 1676174321685 (+12 ms)Cleaning up detritus from prior splits at 1676174321685Running coprocessor post-open hooks at 1676174321689 (+4 ms)Region opened successfully at 1676174321689 2023-02-12 03:58:41,690 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76., pid=47, masterSystemTime=1676174321643 2023-02-12 03:58:41,690 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c., pid=45, masterSystemTime=1676174321642 2023-02-12 03:58:41,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:41,692 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:41,693 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:41,693 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174321693"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174321693"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174321693"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174321693"}]},"ts":"1676174321693"} 2023-02-12 03:58:41,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:41,694 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:41,695 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:41,695 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174321695"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174321695"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174321695"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174321695"}]},"ts":"1676174321695"} 2023-02-12 03:58:41,695 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:41,696 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened b6894e9d4c505cdcd16894575e6b138a; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70561495, jitterRate=0.05144821107387543}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:41,696 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for b6894e9d4c505cdcd16894575e6b138a: Running coprocessor pre-open hook at 1676174321666Writing region info on filesystem at 1676174321666Initializing all the Stores at 1676174321667 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174321667Cleaning up temporary data from old regions at 1676174321691 (+24 ms)Cleaning up detritus from prior splits at 1676174321691Running coprocessor post-open hooks at 1676174321696 (+5 ms)Region opened successfully at 1676174321696 2023-02-12 03:58:41,698 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a., pid=43, masterSystemTime=1676174321639 2023-02-12 03:58:41,699 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=40 2023-02-12 03:58:41,700 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=40, state=SUCCESS; OpenRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451 in 207 msec 2023-02-12 03:58:41,700 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:41,700 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:41,701 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=47, resume processing ppid=41 2023-02-12 03:58:41,701 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=47, ppid=41, state=SUCCESS; OpenRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,42205,1676174315413 in 206 msec 2023-02-12 03:58:41,701 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:41,701 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174321701"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174321701"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174321701"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174321701"}]},"ts":"1676174321701"} 2023-02-12 03:58:41,702 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN in 379 msec 2023-02-12 03:58:41,703 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN in 380 msec 2023-02-12 03:58:41,706 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=42 2023-02-12 03:58:41,706 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=42, state=SUCCESS; OpenRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,35785,1676174315481 in 218 msec 2023-02-12 03:58:41,708 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=36 2023-02-12 03:58:41,708 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN in 385 msec 2023-02-12 03:58:41,709 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174321709"}]},"ts":"1676174321709"} 2023-02-12 03:58:41,711 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=ENABLED in hbase:meta 2023-02-12 03:58:41,712 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676174318078' was successfully enabled. 2023-02-12 03:58:41,716 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=36, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676174318078 in 406 msec 2023-02-12 03:58:42,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-12 03:58:42,069 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676174318078, procId: 36 completed 2023-02-12 03:58:42,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:58:42,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:58:42,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:58:42,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:58:42,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:58:42,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:58:42,122 DEBUG [Listener at localhost.localdomain/46483] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676174318078 2023-02-12 03:58:42,122 DEBUG [Listener at localhost.localdomain/46483] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:42,123 INFO [Listener at localhost.localdomain/46483] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-12 03:58:42,144 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-12 03:58:42,147 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-12 03:58:42,150 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-12 03:58:42,152 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-12 03:58:42,154 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-12 03:58:42,167 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676174318078 2023-02-12 03:58:42,167 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$11(2405): Client=jenkins//148.251.75.209 disable testRestoreSnapshot-1676174318078 2023-02-12 03:58:42,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:42,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-12 03:58:42,172 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174322172"}]},"ts":"1676174322172"} 2023-02-12 03:58:42,174 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=DISABLING in hbase:meta 2023-02-12 03:58:42,175 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676174318078 to state=DISABLING 2023-02-12 03:58:42,176 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, UNASSIGN}, {pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, UNASSIGN}, {pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, UNASSIGN}, {pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, UNASSIGN}, {pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, UNASSIGN}, {pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, UNASSIGN}] 2023-02-12 03:58:42,178 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, UNASSIGN 2023-02-12 03:58:42,178 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, UNASSIGN 2023-02-12 03:58:42,178 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, UNASSIGN 2023-02-12 03:58:42,179 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, UNASSIGN 2023-02-12 03:58:42,179 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, UNASSIGN 2023-02-12 03:58:42,180 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:42,180 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:42,180 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:42,180 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:42,180 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:42,180 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174322180"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174322180"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174322180"}]},"ts":"1676174322180"} 2023-02-12 03:58:42,180 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174322180"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174322180"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174322180"}]},"ts":"1676174322180"} 2023-02-12 03:58:42,180 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174322180"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174322180"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174322180"}]},"ts":"1676174322180"} 2023-02-12 03:58:42,180 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174322180"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174322180"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174322180"}]},"ts":"1676174322180"} 2023-02-12 03:58:42,180 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174322180"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174322180"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174322180"}]},"ts":"1676174322180"} 2023-02-12 03:58:42,182 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=54, state=RUNNABLE; CloseRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:42,183 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=57, ppid=55, state=RUNNABLE; CloseRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:42,184 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=53, state=RUNNABLE; CloseRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:42,186 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=50, state=RUNNABLE; CloseRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:42,188 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=51, state=RUNNABLE; CloseRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:42,189 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, UNASSIGN 2023-02-12 03:58:42,191 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:42,192 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174322191"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174322191"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174322191"}]},"ts":"1676174322191"} 2023-02-12 03:58:42,194 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=52, state=RUNNABLE; CloseRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:42,336 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:42,337 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing e019e3f4c00b3da154d59b4085911d76, disabling compactions & flushes 2023-02-12 03:58:42,338 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:42,338 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:42,338 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. after waiting 0 ms 2023-02-12 03:58:42,338 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:42,339 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing e019e3f4c00b3da154d59b4085911d76 1/1 column families, dataSize=1.70 KB heapSize=3.91 KB 2023-02-12 03:58:42,340 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:42,340 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:42,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing b6894e9d4c505cdcd16894575e6b138a, disabling compactions & flushes 2023-02-12 03:58:42,340 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:42,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:42,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing cc0e0b01138d4eae5df1e5641b484f45, disabling compactions & flushes 2023-02-12 03:58:42,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. after waiting 0 ms 2023-02-12 03:58:42,341 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:42,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:42,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:42,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. after waiting 0 ms 2023-02-12 03:58:42,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:42,341 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing b6894e9d4c505cdcd16894575e6b138a 1/1 column families, dataSize=23.09 KB heapSize=49.89 KB 2023-02-12 03:58:42,341 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing cc0e0b01138d4eae5df1e5641b484f45 1/1 column families, dataSize=1.44 KB heapSize=3.34 KB 2023-02-12 03:58:42,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-12 03:58:42,465 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 2023-02-12 03:58:42,465 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b 2023-02-12 03:58:42,492 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/.tmp/cf/541575adc02c4606b85cbfea9970d250 2023-02-12 03:58:42,492 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=23.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/.tmp/cf/edd4e077572a45bcac8218359e3d1f91 2023-02-12 03:58:42,501 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/.tmp/cf/541575adc02c4606b85cbfea9970d250 as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/541575adc02c4606b85cbfea9970d250 2023-02-12 03:58:42,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/.tmp/cf/edd4e077572a45bcac8218359e3d1f91 as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/edd4e077572a45bcac8218359e3d1f91 2023-02-12 03:58:42,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/edd4e077572a45bcac8218359e3d1f91, entries=353, sequenceid=8, filesize=65.0 K 2023-02-12 03:58:42,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/541575adc02c4606b85cbfea9970d250, entries=26, sequenceid=8, filesize=9.3 K 2023-02-12 03:58:42,513 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~23.09 KB/23647, heapSize ~49.88 KB/51072, currentSize=0 B/0 for b6894e9d4c505cdcd16894575e6b138a in 172ms, sequenceid=8, compaction requested=false 2023-02-12 03:58:42,513 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for e019e3f4c00b3da154d59b4085911d76 in 175ms, sequenceid=8, compaction requested=false 2023-02-12 03:58:42,514 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676174318078' 2023-02-12 03:58:42,514 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676174318078' 2023-02-12 03:58:42,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 03:58:42,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 03:58:42,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:42,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:42,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for b6894e9d4c505cdcd16894575e6b138a: Waiting for close lock at 1676174322340Running coprocessor pre-close hooks at 1676174322340Disabling compacts and flushes for region at 1676174322340Disabling writes for close at 1676174322341 (+1 ms)Obtaining lock to block concurrent updates at 1676174322341Preparing flush snapshotting stores in b6894e9d4c505cdcd16894575e6b138a at 1676174322341Finished memstore snapshotting testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a., syncing WAL and waiting on mvcc, flushsize=dataSize=23647, getHeapSize=51072, getOffHeapSize=0, getCellsCount=353 at 1676174322349 (+8 ms)Flushing stores of testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. at 1676174322353 (+4 ms)Flushing b6894e9d4c505cdcd16894575e6b138a/cf: creating writer at 1676174322356 (+3 ms)Flushing mob file b6894e9d4c505cdcd16894575e6b138a/cf: appending metadata at 1676174322410 (+54 ms)Flushing mob file b6894e9d4c505cdcd16894575e6b138a/cf: closing flushed file at 1676174322410Flushing b6894e9d4c505cdcd16894575e6b138a/cf: appending metadata at 1676174322467 (+57 ms)Flushing b6894e9d4c505cdcd16894575e6b138a/cf: closing flushed file at 1676174322470 (+3 ms)Flushing b6894e9d4c505cdcd16894575e6b138a/cf: reopening flushed file at 1676174322504 (+34 ms)Finished flush of dataSize ~23.09 KB/23647, heapSize ~49.88 KB/51072, currentSize=0 B/0 for b6894e9d4c505cdcd16894575e6b138a in 172ms, sequenceid=8, compaction requested=false at 1676174322513 (+9 ms)Writing region close event to WAL at 1676174322521 (+8 ms)Running coprocessor post-close hooks at 1676174322525 (+4 ms)Closed at 1676174322525 2023-02-12 03:58:42,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for e019e3f4c00b3da154d59b4085911d76: Waiting for close lock at 1676174322337Running coprocessor pre-close hooks at 1676174322337Disabling compacts and flushes for region at 1676174322337Disabling writes for close at 1676174322338 (+1 ms)Obtaining lock to block concurrent updates at 1676174322339 (+1 ms)Preparing flush snapshotting stores in e019e3f4c00b3da154d59b4085911d76 at 1676174322339Finished memstore snapshotting testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76., syncing WAL and waiting on mvcc, flushsize=dataSize=1740, getHeapSize=3984, getOffHeapSize=0, getCellsCount=26 at 1676174322350 (+11 ms)Flushing stores of testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. at 1676174322353 (+3 ms)Flushing e019e3f4c00b3da154d59b4085911d76/cf: creating writer at 1676174322357 (+4 ms)Flushing mob file e019e3f4c00b3da154d59b4085911d76/cf: appending metadata at 1676174322398 (+41 ms)Flushing mob file e019e3f4c00b3da154d59b4085911d76/cf: closing flushed file at 1676174322398Flushing e019e3f4c00b3da154d59b4085911d76/cf: appending metadata at 1676174322467 (+69 ms)Flushing e019e3f4c00b3da154d59b4085911d76/cf: closing flushed file at 1676174322470 (+3 ms)Flushing e019e3f4c00b3da154d59b4085911d76/cf: reopening flushed file at 1676174322503 (+33 ms)Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for e019e3f4c00b3da154d59b4085911d76 in 175ms, sequenceid=8, compaction requested=false at 1676174322513 (+10 ms)Writing region close event to WAL at 1676174322521 (+8 ms)Running coprocessor post-close hooks at 1676174322525 (+4 ms)Closed at 1676174322525 2023-02-12 03:58:42,528 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:42,528 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:42,529 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing ecda59e01589f4f279b19a5420a1e608, disabling compactions & flushes 2023-02-12 03:58:42,529 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=CLOSED 2023-02-12 03:58:42,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:42,529 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:42,529 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. after waiting 0 ms 2023-02-12 03:58:42,529 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174322529"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174322529"}]},"ts":"1676174322529"} 2023-02-12 03:58:42,529 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:42,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:42,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing ecda59e01589f4f279b19a5420a1e608 1/1 column families, dataSize=1.76 KB heapSize=4.05 KB 2023-02-12 03:58:42,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:42,530 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing fcd2ed887eb9f47c07236428879982c1, disabling compactions & flushes 2023-02-12 03:58:42,530 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:42,530 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:42,530 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. after waiting 0 ms 2023-02-12 03:58:42,530 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:42,530 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing fcd2ed887eb9f47c07236428879982c1 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-12 03:58:42,530 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=CLOSED 2023-02-12 03:58:42,531 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174322530"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174322530"}]},"ts":"1676174322530"} 2023-02-12 03:58:42,537 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=54 2023-02-12 03:58:42,537 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=54, state=SUCCESS; CloseRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,42205,1676174315413 in 349 msec 2023-02-12 03:58:42,538 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=57, resume processing ppid=55 2023-02-12 03:58:42,538 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, ppid=55, state=SUCCESS; CloseRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,35785,1676174315481 in 351 msec 2023-02-12 03:58:42,539 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, UNASSIGN in 361 msec 2023-02-12 03:58:42,540 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, UNASSIGN in 362 msec 2023-02-12 03:58:42,555 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-12 03:58:42,570 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c 2023-02-12 03:58:42,586 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/.tmp/cf/f73ec6d83ce444e98c68cfc9bb0662a3 2023-02-12 03:58:42,596 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/.tmp/cf/f73ec6d83ce444e98c68cfc9bb0662a3 as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/f73ec6d83ce444e98c68cfc9bb0662a3 2023-02-12 03:58:42,604 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/f73ec6d83ce444e98c68cfc9bb0662a3, entries=36, sequenceid=8, filesize=11.0 K 2023-02-12 03:58:42,606 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for fcd2ed887eb9f47c07236428879982c1 in 76ms, sequenceid=8, compaction requested=false 2023-02-12 03:58:42,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 03:58:42,618 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:42,618 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for fcd2ed887eb9f47c07236428879982c1: Waiting for close lock at 1676174322529Running coprocessor pre-close hooks at 1676174322529Disabling compacts and flushes for region at 1676174322529Disabling writes for close at 1676174322530 (+1 ms)Obtaining lock to block concurrent updates at 1676174322530Preparing flush snapshotting stores in fcd2ed887eb9f47c07236428879982c1 at 1676174322530Finished memstore snapshotting testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1., syncing WAL and waiting on mvcc, flushsize=dataSize=2410, getHeapSize=5424, getOffHeapSize=0, getCellsCount=36 at 1676174322531 (+1 ms)Flushing stores of testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. at 1676174322531Flushing fcd2ed887eb9f47c07236428879982c1/cf: creating writer at 1676174322532 (+1 ms)Flushing mob file fcd2ed887eb9f47c07236428879982c1/cf: appending metadata at 1676174322540 (+8 ms)Flushing mob file fcd2ed887eb9f47c07236428879982c1/cf: closing flushed file at 1676174322540Flushing fcd2ed887eb9f47c07236428879982c1/cf: appending metadata at 1676174322572 (+32 ms)Flushing fcd2ed887eb9f47c07236428879982c1/cf: closing flushed file at 1676174322572Flushing fcd2ed887eb9f47c07236428879982c1/cf: reopening flushed file at 1676174322597 (+25 ms)Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for fcd2ed887eb9f47c07236428879982c1 in 76ms, sequenceid=8, compaction requested=false at 1676174322607 (+10 ms)Writing region close event to WAL at 1676174322613 (+6 ms)Running coprocessor post-close hooks at 1676174322618 (+5 ms)Closed at 1676174322618 2023-02-12 03:58:42,630 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:42,631 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=CLOSED 2023-02-12 03:58:42,631 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174322631"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174322631"}]},"ts":"1676174322631"} 2023-02-12 03:58:42,636 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=52 2023-02-12 03:58:42,636 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=52, state=SUCCESS; CloseRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,35785,1676174315481 in 439 msec 2023-02-12 03:58:42,638 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, UNASSIGN in 460 msec 2023-02-12 03:58:42,654 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-12 03:58:42,655 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-12 03:58:42,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b 2023-02-12 03:58:42,864 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/.tmp/cf/027aff06c1054aaaa903715be8404fab 2023-02-12 03:58:42,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/.tmp/cf/027aff06c1054aaaa903715be8404fab as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/027aff06c1054aaaa903715be8404fab 2023-02-12 03:58:42,887 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/027aff06c1054aaaa903715be8404fab, entries=22, sequenceid=8, filesize=8.6 K 2023-02-12 03:58:42,890 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.44 KB/1472, heapSize ~3.33 KB/3408, currentSize=0 B/0 for cc0e0b01138d4eae5df1e5641b484f45 in 548ms, sequenceid=8, compaction requested=false 2023-02-12 03:58:42,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 03:58:42,899 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:42,899 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for cc0e0b01138d4eae5df1e5641b484f45: Waiting for close lock at 1676174322340Running coprocessor pre-close hooks at 1676174322340Disabling compacts and flushes for region at 1676174322340Disabling writes for close at 1676174322341 (+1 ms)Obtaining lock to block concurrent updates at 1676174322341Preparing flush snapshotting stores in cc0e0b01138d4eae5df1e5641b484f45 at 1676174322341Finished memstore snapshotting testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45., syncing WAL and waiting on mvcc, flushsize=dataSize=1472, getHeapSize=3408, getOffHeapSize=0, getCellsCount=22 at 1676174322349 (+8 ms)Flushing stores of testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. at 1676174322353 (+4 ms)Flushing cc0e0b01138d4eae5df1e5641b484f45/cf: creating writer at 1676174322356 (+3 ms)Flushing mob file cc0e0b01138d4eae5df1e5641b484f45/cf: appending metadata at 1676174322397 (+41 ms)Flushing mob file cc0e0b01138d4eae5df1e5641b484f45/cf: closing flushed file at 1676174322397Flushing cc0e0b01138d4eae5df1e5641b484f45/cf: appending metadata at 1676174322841 (+444 ms)Flushing cc0e0b01138d4eae5df1e5641b484f45/cf: closing flushed file at 1676174322841Flushing cc0e0b01138d4eae5df1e5641b484f45/cf: reopening flushed file at 1676174322878 (+37 ms)Finished flush of dataSize ~1.44 KB/1472, heapSize ~3.33 KB/3408, currentSize=0 B/0 for cc0e0b01138d4eae5df1e5641b484f45 in 548ms, sequenceid=8, compaction requested=false at 1676174322890 (+12 ms)Writing region close event to WAL at 1676174322892 (+2 ms)Running coprocessor post-close hooks at 1676174322898 (+6 ms)Closed at 1676174322899 (+1 ms) 2023-02-12 03:58:42,901 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:42,901 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:42,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 609b1a331ab507675f39b4bdbe7a382c, disabling compactions & flushes 2023-02-12 03:58:42,901 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:42,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:42,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. after waiting 0 ms 2023-02-12 03:58:42,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:42,902 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 609b1a331ab507675f39b4bdbe7a382c 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-12 03:58:42,902 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=CLOSED 2023-02-12 03:58:42,902 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174322902"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174322902"}]},"ts":"1676174322902"} 2023-02-12 03:58:42,909 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=50 2023-02-12 03:58:42,909 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=50, state=SUCCESS; CloseRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35783,1676174315451 in 719 msec 2023-02-12 03:58:42,911 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, UNASSIGN in 733 msec 2023-02-12 03:58:42,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-12 03:58:42,938 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 2023-02-12 03:58:42,964 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/.tmp/cf/de3f19ac4a9247fbac10bbe6eed0d5d1 2023-02-12 03:58:42,967 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 2023-02-12 03:58:42,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/.tmp/cf/de3f19ac4a9247fbac10bbe6eed0d5d1 as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/de3f19ac4a9247fbac10bbe6eed0d5d1 2023-02-12 03:58:42,980 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/de3f19ac4a9247fbac10bbe6eed0d5d1, entries=36, sequenceid=8, filesize=11.0 K 2023-02-12 03:58:42,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 609b1a331ab507675f39b4bdbe7a382c in 80ms, sequenceid=8, compaction requested=false 2023-02-12 03:58:42,986 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/.tmp/cf/f75b9c751c584507ac2da36d4b2eb053 2023-02-12 03:58:43,000 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 03:58:43,002 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:43,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 609b1a331ab507675f39b4bdbe7a382c: Waiting for close lock at 1676174322901Running coprocessor pre-close hooks at 1676174322901Disabling compacts and flushes for region at 1676174322901Disabling writes for close at 1676174322902 (+1 ms)Obtaining lock to block concurrent updates at 1676174322902Preparing flush snapshotting stores in 609b1a331ab507675f39b4bdbe7a382c at 1676174322902Finished memstore snapshotting testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c., syncing WAL and waiting on mvcc, flushsize=dataSize=2410, getHeapSize=5424, getOffHeapSize=0, getCellsCount=36 at 1676174322902Flushing stores of testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. at 1676174322902Flushing 609b1a331ab507675f39b4bdbe7a382c/cf: creating writer at 1676174322904 (+2 ms)Flushing mob file 609b1a331ab507675f39b4bdbe7a382c/cf: appending metadata at 1676174322914 (+10 ms)Flushing mob file 609b1a331ab507675f39b4bdbe7a382c/cf: closing flushed file at 1676174322914Flushing 609b1a331ab507675f39b4bdbe7a382c/cf: appending metadata at 1676174322941 (+27 ms)Flushing 609b1a331ab507675f39b4bdbe7a382c/cf: closing flushed file at 1676174322941Flushing 609b1a331ab507675f39b4bdbe7a382c/cf: reopening flushed file at 1676174322973 (+32 ms)Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 609b1a331ab507675f39b4bdbe7a382c in 80ms, sequenceid=8, compaction requested=false at 1676174322983 (+10 ms)Writing region close event to WAL at 1676174322996 (+13 ms)Running coprocessor post-close hooks at 1676174323001 (+5 ms)Closed at 1676174323002 (+1 ms) 2023-02-12 03:58:43,005 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:43,005 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=CLOSED 2023-02-12 03:58:43,005 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/.tmp/cf/f75b9c751c584507ac2da36d4b2eb053 as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/f75b9c751c584507ac2da36d4b2eb053 2023-02-12 03:58:43,005 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174323005"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174323005"}]},"ts":"1676174323005"} 2023-02-12 03:58:43,011 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=53 2023-02-12 03:58:43,011 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=53, state=SUCCESS; CloseRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451 in 823 msec 2023-02-12 03:58:43,013 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=53, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, UNASSIGN in 835 msec 2023-02-12 03:58:43,014 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/f75b9c751c584507ac2da36d4b2eb053, entries=27, sequenceid=8, filesize=9.5 K 2023-02-12 03:58:43,016 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for ecda59e01589f4f279b19a5420a1e608 in 487ms, sequenceid=8, compaction requested=false 2023-02-12 03:58:43,024 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 03:58:43,026 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:43,026 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for ecda59e01589f4f279b19a5420a1e608: Waiting for close lock at 1676174322528Running coprocessor pre-close hooks at 1676174322528Disabling compacts and flushes for region at 1676174322528Disabling writes for close at 1676174322529 (+1 ms)Obtaining lock to block concurrent updates at 1676174322529Preparing flush snapshotting stores in ecda59e01589f4f279b19a5420a1e608 at 1676174322529Finished memstore snapshotting testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608., syncing WAL and waiting on mvcc, flushsize=dataSize=1807, getHeapSize=4128, getOffHeapSize=0, getCellsCount=27 at 1676174322530 (+1 ms)Flushing stores of testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. at 1676174322533 (+3 ms)Flushing ecda59e01589f4f279b19a5420a1e608/cf: creating writer at 1676174322533Flushing mob file ecda59e01589f4f279b19a5420a1e608/cf: appending metadata at 1676174322540 (+7 ms)Flushing mob file ecda59e01589f4f279b19a5420a1e608/cf: closing flushed file at 1676174322540Flushing ecda59e01589f4f279b19a5420a1e608/cf: appending metadata at 1676174322969 (+429 ms)Flushing ecda59e01589f4f279b19a5420a1e608/cf: closing flushed file at 1676174322970 (+1 ms)Flushing ecda59e01589f4f279b19a5420a1e608/cf: reopening flushed file at 1676174323006 (+36 ms)Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for ecda59e01589f4f279b19a5420a1e608 in 487ms, sequenceid=8, compaction requested=false at 1676174323016 (+10 ms)Writing region close event to WAL at 1676174323018 (+2 ms)Running coprocessor post-close hooks at 1676174323025 (+7 ms)Closed at 1676174323026 (+1 ms) 2023-02-12 03:58:43,028 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:43,029 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=CLOSED 2023-02-12 03:58:43,029 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174323029"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174323029"}]},"ts":"1676174323029"} 2023-02-12 03:58:43,033 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=51 2023-02-12 03:58:43,033 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=51, state=SUCCESS; CloseRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,42205,1676174315413 in 842 msec 2023-02-12 03:58:43,036 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=49 2023-02-12 03:58:43,036 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, UNASSIGN in 857 msec 2023-02-12 03:58:43,038 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174323038"}]},"ts":"1676174323038"} 2023-02-12 03:58:43,040 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=DISABLED in hbase:meta 2023-02-12 03:58:43,041 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676174318078 to state=DISABLED 2023-02-12 03:58:43,047 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676174318078 in 877 msec 2023-02-12 03:58:43,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-12 03:58:43,682 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676174318078, procId: 49 completed 2023-02-12 03:58:43,682 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1580): Client=jenkins//148.251.75.209 snapshot request for:{ ss=snaptb0-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } 2023-02-12 03:58:43,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676174323683 (current time:1676174323683). 2023-02-12 03:58:43,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-12 03:58:43,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] zookeeper.ReadOnlyZKClient(139): Connect 0x7781605c to 127.0.0.1:50368 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 03:58:43,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4029bd8a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 03:58:43,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:43,692 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47318, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:43,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7781605c to 127.0.0.1:50368 2023-02-12 03:58:43,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:58:43,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-12 03:58:43,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } 2023-02-12 03:58:43,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE 2023-02-12 03:58:43,697 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE 2023-02-12 03:58:43,698 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=62, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE; timeout=600000, timestamp=1676174923698 2023-02-12 03:58:43,698 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676174318078 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676174318078 2023-02-12 03:58:43,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } 2023-02-12 03:58:43,699 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-12 03:58:43,699 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-12 03:58:43,717 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676174318078 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } 2023-02-12 03:58:43,719 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:43,719 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:43,719 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:43,719 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:43,720 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:43,720 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:43,720 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:43,720 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:43,722 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:43,720 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:43,720 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:43,720 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:43,722 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:43,722 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:43,722 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/027aff06c1054aaaa903715be8404fab] hfiles 2023-02-12 03:58:43,723 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/027aff06c1054aaaa903715be8404fab 2023-02-12 03:58:43,722 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/541575adc02c4606b85cbfea9970d250] hfiles 2023-02-12 03:58:43,723 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/541575adc02c4606b85cbfea9970d250 2023-02-12 03:58:43,730 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/f75b9c751c584507ac2da36d4b2eb053] hfiles 2023-02-12 03:58:43,730 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/f75b9c751c584507ac2da36d4b2eb053 2023-02-12 03:58:43,731 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/f73ec6d83ce444e98c68cfc9bb0662a3] hfiles 2023-02-12 03:58:43,731 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/f73ec6d83ce444e98c68cfc9bb0662a3 2023-02-12 03:58:43,732 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/edd4e077572a45bcac8218359e3d1f91] hfiles 2023-02-12 03:58:43,732 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/de3f19ac4a9247fbac10bbe6eed0d5d1] hfiles 2023-02-12 03:58:43,732 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/edd4e077572a45bcac8218359e3d1f91 2023-02-12 03:58:43,732 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/de3f19ac4a9247fbac10bbe6eed0d5d1 2023-02-12 03:58:43,735 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28] hfiles 2023-02-12 03:58:43,735 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b 2023-02-12 03:58:43,735 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 2023-02-12 03:58:43,739 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c 2023-02-12 03:58:43,739 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b 2023-02-12 03:58:43,739 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 2023-02-12 03:58:43,739 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 2023-02-12 03:58:43,805 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676174318078 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } as finished. 2023-02-12 03:58:43,805 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676174318078 2023-02-12 03:58:43,806 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/snaptb0-1676174318078 2023-02-12 03:58:43,859 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/snaptb0-1676174318078 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/snaptb0-1676174318078 2023-02-12 03:58:43,889 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676174318078 of table testRestoreSnapshot-1676174318078 completed 2023-02-12 03:58:43,889 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/snaptb0-1676174318078 2023-02-12 03:58:43,889 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/snaptb0-1676174318078 2023-02-12 03:58:43,889 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676174318078 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676174318078 at 1676174323698Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676174318078 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } at 1676174323717 (+19 ms)Consolidate snapshot: snaptb0-1676174318078 at 1676174323805 (+88 ms)Loading Region manifests for snaptb0-1676174318078 at 1676174323806 (+1 ms)Writing data manifest for snaptb0-1676174318078 at 1676174323820 (+14 ms)Verifying snapshot: snaptb0-1676174318078 at 1676174323841 (+21 ms)Snapshot snaptb0-1676174318078 of table testRestoreSnapshot-1676174318078 completed at 1676174323889 (+48 ms) 2023-02-12 03:58:43,891 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=62, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE 2023-02-12 03:58:43,892 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE in 196 msec 2023-02-12 03:58:43,949 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 03:58:43,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } is done 2023-02-12 03:58:43,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-12 03:58:43,951 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676174318078 2023-02-12 03:58:43,952 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$10(2372): Client=jenkins//148.251.75.209 enable testRestoreSnapshot-1676174318078 2023-02-12 03:58:43,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:43,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-12 03:58:43,957 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676174318078 2023-02-12 03:58:43,957 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174323957"}]},"ts":"1676174323957"} 2023-02-12 03:58:43,960 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=ENABLING in hbase:meta 2023-02-12 03:58:43,963 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-12 03:58:43,964 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN}, {pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN}, {pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN}, {pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN}, {pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN}, {pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN}] 2023-02-12 03:58:43,966 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN 2023-02-12 03:58:43,966 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN 2023-02-12 03:58:43,966 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN 2023-02-12 03:58:43,966 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN 2023-02-12 03:58:43,967 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:43,967 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:43,967 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,42205,1676174315413; forceNewPlan=false, retain=true 2023-02-12 03:58:43,967 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,35783,1676174315451; forceNewPlan=false, retain=true 2023-02-12 03:58:43,967 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:43,968 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,35785,1676174315481; forceNewPlan=false, retain=true 2023-02-12 03:58:43,968 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:43,968 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,35783,1676174315451; forceNewPlan=false, retain=true 2023-02-12 03:58:43,968 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN 2023-02-12 03:58:43,969 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN 2023-02-12 03:58:43,980 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:43,980 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,35785,1676174315481; forceNewPlan=false, retain=true 2023-02-12 03:58:43,980 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:43,981 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,42205,1676174315413; forceNewPlan=false, retain=true 2023-02-12 03:58:44,118 INFO [jenkins-hbase20:37305] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-12 03:58:44,118 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:44,118 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:44,118 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:44,118 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174324118"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174324118"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174324118"}]},"ts":"1676174324118"} 2023-02-12 03:58:44,118 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174324118"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174324118"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174324118"}]},"ts":"1676174324118"} 2023-02-12 03:58:44,118 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174324118"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174324118"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174324118"}]},"ts":"1676174324118"} 2023-02-12 03:58:44,118 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:44,118 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:44,119 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174324118"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174324118"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174324118"}]},"ts":"1676174324118"} 2023-02-12 03:58:44,119 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174324118"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174324118"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174324118"}]},"ts":"1676174324118"} 2023-02-12 03:58:44,121 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=70, ppid=67, state=RUNNABLE; OpenRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:44,122 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=71, ppid=65, state=RUNNABLE; OpenRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:44,124 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=72, ppid=66, state=RUNNABLE; OpenRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:44,126 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:44,126 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174324126"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174324126"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174324126"}]},"ts":"1676174324126"} 2023-02-12 03:58:44,134 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=69, state=RUNNABLE; OpenRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:44,135 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=74, ppid=68, state=RUNNABLE; OpenRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:44,138 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=75, ppid=64, state=RUNNABLE; OpenRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:44,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-12 03:58:44,278 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:44,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => cc0e0b01138d4eae5df1e5641b484f45, NAME => 'testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.', STARTKEY => '', ENDKEY => '1'} 2023-02-12 03:58:44,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:44,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:44,279 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:44,279 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:44,280 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:44,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b6894e9d4c505cdcd16894575e6b138a, NAME => 'testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.', STARTKEY => '5', ENDKEY => ''} 2023-02-12 03:58:44,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:44,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:44,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:44,281 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:44,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e019e3f4c00b3da154d59b4085911d76, NAME => 'testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.', STARTKEY => '4', ENDKEY => '5'} 2023-02-12 03:58:44,281 INFO [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:44,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:44,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:44,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:44,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:44,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:44,282 DEBUG [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf 2023-02-12 03:58:44,282 DEBUG [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf 2023-02-12 03:58:44,283 INFO [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cc0e0b01138d4eae5df1e5641b484f45 columnFamilyName cf 2023-02-12 03:58:44,283 INFO [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:44,283 INFO [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:44,284 DEBUG [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf 2023-02-12 03:58:44,284 DEBUG [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf 2023-02-12 03:58:44,284 DEBUG [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf 2023-02-12 03:58:44,284 DEBUG [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf 2023-02-12 03:58:44,285 INFO [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e019e3f4c00b3da154d59b4085911d76 columnFamilyName cf 2023-02-12 03:58:44,285 INFO [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b6894e9d4c505cdcd16894575e6b138a columnFamilyName cf 2023-02-12 03:58:44,299 DEBUG [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/027aff06c1054aaaa903715be8404fab 2023-02-12 03:58:44,302 INFO [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] regionserver.HStore(310): Store=cc0e0b01138d4eae5df1e5641b484f45/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:44,303 DEBUG [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/edd4e077572a45bcac8218359e3d1f91 2023-02-12 03:58:44,303 INFO [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] regionserver.HStore(310): Store=b6894e9d4c505cdcd16894575e6b138a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:44,304 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:44,304 DEBUG [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/541575adc02c4606b85cbfea9970d250 2023-02-12 03:58:44,304 INFO [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] regionserver.HStore(310): Store=e019e3f4c00b3da154d59b4085911d76/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:44,304 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:44,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:44,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:44,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:44,307 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:44,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:44,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:44,311 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened cc0e0b01138d4eae5df1e5641b484f45; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74043494, jitterRate=0.10333403944969177}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:44,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for cc0e0b01138d4eae5df1e5641b484f45: Running coprocessor pre-open hook at 1676174324279Writing region info on filesystem at 1676174324279Initializing all the Stores at 1676174324279Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174324280 (+1 ms)Cleaning up temporary data from old regions at 1676174324306 (+26 ms)Cleaning up detritus from prior splits at 1676174324307 (+1 ms)Running coprocessor post-open hooks at 1676174324311 (+4 ms)Region opened successfully at 1676174324311 2023-02-12 03:58:44,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:44,312 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45., pid=75, masterSystemTime=1676174324274 2023-02-12 03:58:44,312 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened b6894e9d4c505cdcd16894575e6b138a; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71307279, jitterRate=0.06256125867366791}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:44,312 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for b6894e9d4c505cdcd16894575e6b138a: Running coprocessor pre-open hook at 1676174324281Writing region info on filesystem at 1676174324281Initializing all the Stores at 1676174324283 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174324283Cleaning up temporary data from old regions at 1676174324306 (+23 ms)Cleaning up detritus from prior splits at 1676174324307 (+1 ms)Running coprocessor post-open hooks at 1676174324312 (+5 ms)Region opened successfully at 1676174324312 2023-02-12 03:58:44,313 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened e019e3f4c00b3da154d59b4085911d76; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75403532, jitterRate=0.12360018491744995}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:44,313 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for e019e3f4c00b3da154d59b4085911d76: Running coprocessor pre-open hook at 1676174324281Writing region info on filesystem at 1676174324281Initializing all the Stores at 1676174324282 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174324282Cleaning up temporary data from old regions at 1676174324308 (+26 ms)Cleaning up detritus from prior splits at 1676174324309 (+1 ms)Running coprocessor post-open hooks at 1676174324313 (+4 ms)Region opened successfully at 1676174324313 2023-02-12 03:58:44,313 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a., pid=73, masterSystemTime=1676174324276 2023-02-12 03:58:44,314 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76., pid=74, masterSystemTime=1676174324276 2023-02-12 03:58:44,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:44,314 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:44,315 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:44,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 609b1a331ab507675f39b4bdbe7a382c, NAME => 'testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.', STARTKEY => '3', ENDKEY => '4'} 2023-02-12 03:58:44,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:44,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:44,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:44,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:44,315 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:44,316 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174324315"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174324315"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174324315"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174324315"}]},"ts":"1676174324315"} 2023-02-12 03:58:44,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:44,316 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:44,316 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:44,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => fcd2ed887eb9f47c07236428879982c1, NAME => 'testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.', STARTKEY => '2', ENDKEY => '3'} 2023-02-12 03:58:44,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:44,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:44,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:44,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:44,318 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:44,318 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174324318"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174324318"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174324318"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174324318"}]},"ts":"1676174324318"} 2023-02-12 03:58:44,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:44,318 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:44,318 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:44,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ecda59e01589f4f279b19a5420a1e608, NAME => 'testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.', STARTKEY => '1', ENDKEY => '2'} 2023-02-12 03:58:44,319 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:44,319 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:44,319 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:44,319 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174324319"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174324319"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174324319"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174324319"}]},"ts":"1676174324319"} 2023-02-12 03:58:44,319 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:44,319 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:44,320 INFO [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:44,320 INFO [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:44,323 DEBUG [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf 2023-02-12 03:58:44,323 DEBUG [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf 2023-02-12 03:58:44,323 DEBUG [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf 2023-02-12 03:58:44,323 DEBUG [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf 2023-02-12 03:58:44,323 INFO [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 609b1a331ab507675f39b4bdbe7a382c columnFamilyName cf 2023-02-12 03:58:44,323 INFO [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fcd2ed887eb9f47c07236428879982c1 columnFamilyName cf 2023-02-12 03:58:44,324 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=75, resume processing ppid=64 2023-02-12 03:58:44,324 INFO [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:44,324 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=75, ppid=64, state=SUCCESS; OpenRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35783,1676174315451 in 183 msec 2023-02-12 03:58:44,326 DEBUG [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf 2023-02-12 03:58:44,326 DEBUG [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf 2023-02-12 03:58:44,326 INFO [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ecda59e01589f4f279b19a5420a1e608 columnFamilyName cf 2023-02-12 03:58:44,327 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=69 2023-02-12 03:58:44,327 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=69, state=SUCCESS; OpenRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,35785,1676174315481 in 186 msec 2023-02-12 03:58:44,329 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=74, resume processing ppid=68 2023-02-12 03:58:44,329 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN in 360 msec 2023-02-12 03:58:44,329 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=74, ppid=68, state=SUCCESS; OpenRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,42205,1676174315413 in 187 msec 2023-02-12 03:58:44,330 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=69, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN in 363 msec 2023-02-12 03:58:44,330 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=68, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN in 365 msec 2023-02-12 03:58:44,332 DEBUG [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/de3f19ac4a9247fbac10bbe6eed0d5d1 2023-02-12 03:58:44,333 DEBUG [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/f73ec6d83ce444e98c68cfc9bb0662a3 2023-02-12 03:58:44,333 INFO [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] regionserver.HStore(310): Store=609b1a331ab507675f39b4bdbe7a382c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:44,333 INFO [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] regionserver.HStore(310): Store=fcd2ed887eb9f47c07236428879982c1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:44,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:44,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:44,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:44,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:44,338 DEBUG [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/f75b9c751c584507ac2da36d4b2eb053 2023-02-12 03:58:44,338 INFO [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] regionserver.HStore(310): Store=ecda59e01589f4f279b19a5420a1e608/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:44,339 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:44,339 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:44,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:44,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 609b1a331ab507675f39b4bdbe7a382c; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64251375, jitterRate=-0.042579904198646545}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:44,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened fcd2ed887eb9f47c07236428879982c1; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64185522, jitterRate=-0.043561190366744995}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:44,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 609b1a331ab507675f39b4bdbe7a382c: Running coprocessor pre-open hook at 1676174324315Writing region info on filesystem at 1676174324315Initializing all the Stores at 1676174324317 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174324317Cleaning up temporary data from old regions at 1676174324335 (+18 ms)Cleaning up detritus from prior splits at 1676174324336 (+1 ms)Running coprocessor post-open hooks at 1676174324340 (+4 ms)Region opened successfully at 1676174324340 2023-02-12 03:58:44,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for fcd2ed887eb9f47c07236428879982c1: Running coprocessor pre-open hook at 1676174324317Writing region info on filesystem at 1676174324317Initializing all the Stores at 1676174324319 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174324319Cleaning up temporary data from old regions at 1676174324336 (+17 ms)Cleaning up detritus from prior splits at 1676174324336Running coprocessor post-open hooks at 1676174324340 (+4 ms)Region opened successfully at 1676174324340 2023-02-12 03:58:44,341 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c., pid=70, masterSystemTime=1676174324274 2023-02-12 03:58:44,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:44,341 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1., pid=72, masterSystemTime=1676174324276 2023-02-12 03:58:44,344 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:44,344 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:44,344 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:44,344 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174324344"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174324344"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174324344"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174324344"}]},"ts":"1676174324344"} 2023-02-12 03:58:44,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:44,345 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:44,346 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:44,346 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174324346"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174324346"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174324346"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174324346"}]},"ts":"1676174324346"} 2023-02-12 03:58:44,347 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:44,349 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened ecda59e01589f4f279b19a5420a1e608; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64970287, jitterRate=-0.03186728060245514}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:44,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for ecda59e01589f4f279b19a5420a1e608: Running coprocessor pre-open hook at 1676174324319Writing region info on filesystem at 1676174324319Initializing all the Stores at 1676174324324 (+5 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174324324Cleaning up temporary data from old regions at 1676174324342 (+18 ms)Cleaning up detritus from prior splits at 1676174324343 (+1 ms)Running coprocessor post-open hooks at 1676174324349 (+6 ms)Region opened successfully at 1676174324349 2023-02-12 03:58:44,351 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608., pid=71, masterSystemTime=1676174324276 2023-02-12 03:58:44,351 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=70, resume processing ppid=67 2023-02-12 03:58:44,352 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=70, ppid=67, state=SUCCESS; OpenRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451 in 226 msec 2023-02-12 03:58:44,354 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=72, resume processing ppid=66 2023-02-12 03:58:44,354 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=67, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN in 388 msec 2023-02-12 03:58:44,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:44,354 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:44,354 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=72, ppid=66, state=SUCCESS; OpenRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,35785,1676174315481 in 225 msec 2023-02-12 03:58:44,355 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:44,355 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174324355"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174324355"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174324355"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174324355"}]},"ts":"1676174324355"} 2023-02-12 03:58:44,356 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=66, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN in 390 msec 2023-02-12 03:58:44,361 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=71, resume processing ppid=65 2023-02-12 03:58:44,361 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=71, ppid=65, state=SUCCESS; OpenRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,42205,1676174315413 in 235 msec 2023-02-12 03:58:44,364 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=65, resume processing ppid=63 2023-02-12 03:58:44,364 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN in 397 msec 2023-02-12 03:58:44,364 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174324364"}]},"ts":"1676174324364"} 2023-02-12 03:58:44,366 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=ENABLED in hbase:meta 2023-02-12 03:58:44,368 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676174318078' was successfully enabled. 2023-02-12 03:58:44,371 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676174318078 in 417 msec 2023-02-12 03:58:44,585 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-12 03:58:44,586 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-12 03:58:44,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-12 03:58:44,709 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676174318078, procId: 63 completed 2023-02-12 03:58:44,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:58:44,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:58:44,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:58:44,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:58:44,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:58:44,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:58:44,741 DEBUG [Listener at localhost.localdomain/46483] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676174318078 2023-02-12 03:58:44,741 DEBUG [Listener at localhost.localdomain/46483] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:44,743 INFO [Listener at localhost.localdomain/46483] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-12 03:58:44,995 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-12 03:58:45,168 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-12 03:58:45,382 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-12 03:58:45,576 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-12 03:58:45,750 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-12 03:58:47,493 DEBUG [Listener at localhost.localdomain/46483] zookeeper.ReadOnlyZKClient(139): Connect 0x36a3b58e to 127.0.0.1:50368 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 03:58:47,499 DEBUG [Listener at localhost.localdomain/46483] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13fd136e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 03:58:47,501 DEBUG [hconnection-0x6ef78f31-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:47,505 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38190, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:47,617 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-12 03:58:47,621 DEBUG [hconnection-0x6ef78f31-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:47,627 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50822, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:47,747 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-12 03:58:47,751 DEBUG [hconnection-0x6ef78f31-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:47,752 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38630, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:47,897 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-12 03:58:48,029 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-12 03:58:48,094 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-12 03:58:48,232 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-12 03:58:49,828 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676174318078 2023-02-12 03:58:49,829 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$11(2405): Client=jenkins//148.251.75.209 disable testRestoreSnapshot-1676174318078 2023-02-12 03:58:49,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=76, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:49,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-12 03:58:49,833 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174329833"}]},"ts":"1676174329833"} 2023-02-12 03:58:49,834 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=DISABLING in hbase:meta 2023-02-12 03:58:49,835 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676174318078 to state=DISABLING 2023-02-12 03:58:49,836 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, UNASSIGN}, {pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, UNASSIGN}, {pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, UNASSIGN}, {pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, UNASSIGN}, {pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, UNASSIGN}, {pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, UNASSIGN}] 2023-02-12 03:58:49,838 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, UNASSIGN 2023-02-12 03:58:49,838 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, UNASSIGN 2023-02-12 03:58:49,838 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, UNASSIGN 2023-02-12 03:58:49,838 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, UNASSIGN 2023-02-12 03:58:49,838 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, UNASSIGN 2023-02-12 03:58:49,839 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:49,839 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:49,839 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174329839"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174329839"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174329839"}]},"ts":"1676174329839"} 2023-02-12 03:58:49,839 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174329839"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174329839"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174329839"}]},"ts":"1676174329839"} 2023-02-12 03:58:49,840 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:49,840 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174329840"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174329840"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174329840"}]},"ts":"1676174329840"} 2023-02-12 03:58:49,840 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:49,840 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:49,840 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174329839"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174329839"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174329839"}]},"ts":"1676174329839"} 2023-02-12 03:58:49,840 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174329839"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174329839"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174329839"}]},"ts":"1676174329839"} 2023-02-12 03:58:49,841 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=83, ppid=82, state=RUNNABLE; CloseRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:49,842 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=84, ppid=81, state=RUNNABLE; CloseRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:49,843 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=85, ppid=79, state=RUNNABLE; CloseRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:49,845 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=86, ppid=80, state=RUNNABLE; CloseRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:49,845 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, UNASSIGN 2023-02-12 03:58:49,846 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=87, ppid=78, state=RUNNABLE; CloseRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:49,847 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:49,847 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174329847"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174329847"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174329847"}]},"ts":"1676174329847"} 2023-02-12 03:58:49,849 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=88, ppid=77, state=RUNNABLE; CloseRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:49,994 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:49,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing b6894e9d4c505cdcd16894575e6b138a, disabling compactions & flushes 2023-02-12 03:58:49,996 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:49,996 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:49,996 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. after waiting 0 ms 2023-02-12 03:58:49,996 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:49,996 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing b6894e9d4c505cdcd16894575e6b138a 1/1 column families, dataSize=22.05 KB heapSize=47.64 KB 2023-02-12 03:58:49,996 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:49,998 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing e019e3f4c00b3da154d59b4085911d76, disabling compactions & flushes 2023-02-12 03:58:49,998 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:49,998 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:49,998 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. after waiting 0 ms 2023-02-12 03:58:49,998 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:49,998 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing e019e3f4c00b3da154d59b4085911d76 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-12 03:58:49,998 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:49,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing cc0e0b01138d4eae5df1e5641b484f45, disabling compactions & flushes 2023-02-12 03:58:49,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:49,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:50,000 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. after waiting 0 ms 2023-02-12 03:58:50,000 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:50,000 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing cc0e0b01138d4eae5df1e5641b484f45 1/1 column families, dataSize=1.96 KB heapSize=4.47 KB 2023-02-12 03:58:50,030 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 2023-02-12 03:58:50,033 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f 2023-02-12 03:58:50,033 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 2023-02-12 03:58:50,040 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/.tmp/cf/10e6966d5cc341688a6163ff0d1d3493 2023-02-12 03:58:50,044 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=22.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/.tmp/cf/6ae5fbf6b3ca4a899b9e29e73711608c 2023-02-12 03:58:50,044 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/.tmp/cf/6148ef2edb69420e9208fff789b259f0 2023-02-12 03:58:50,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/.tmp/cf/10e6966d5cc341688a6163ff0d1d3493 as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/10e6966d5cc341688a6163ff0d1d3493 2023-02-12 03:58:50,049 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/.tmp/cf/6148ef2edb69420e9208fff789b259f0 as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/6148ef2edb69420e9208fff789b259f0 2023-02-12 03:58:50,050 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/.tmp/cf/6ae5fbf6b3ca4a899b9e29e73711608c as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/6ae5fbf6b3ca4a899b9e29e73711608c 2023-02-12 03:58:50,050 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/10e6966d5cc341688a6163ff0d1d3493, entries=30, sequenceid=15, filesize=10.0 K 2023-02-12 03:58:50,051 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for cc0e0b01138d4eae5df1e5641b484f45 in 51ms, sequenceid=15, compaction requested=false 2023-02-12 03:58:50,054 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/6ae5fbf6b3ca4a899b9e29e73711608c, entries=337, sequenceid=15, filesize=62.3 K 2023-02-12 03:58:50,054 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/6148ef2edb69420e9208fff789b259f0, entries=31, sequenceid=15, filesize=10.2 K 2023-02-12 03:58:50,055 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.05 KB/22575, heapSize ~47.63 KB/48768, currentSize=0 B/0 for b6894e9d4c505cdcd16894575e6b138a in 59ms, sequenceid=15, compaction requested=false 2023-02-12 03:58:50,055 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for e019e3f4c00b3da154d59b4085911d76 in 57ms, sequenceid=15, compaction requested=false 2023-02-12 03:58:50,066 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 03:58:50,066 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 03:58:50,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 03:58:50,068 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:50,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for e019e3f4c00b3da154d59b4085911d76: Waiting for close lock at 1676174329996Running coprocessor pre-close hooks at 1676174329996Disabling compacts and flushes for region at 1676174329996Disabling writes for close at 1676174329998 (+2 ms)Obtaining lock to block concurrent updates at 1676174329998Preparing flush snapshotting stores in e019e3f4c00b3da154d59b4085911d76 at 1676174329998Finished memstore snapshotting testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76., syncing WAL and waiting on mvcc, flushsize=dataSize=2075, getHeapSize=4704, getOffHeapSize=0, getCellsCount=31 at 1676174329999 (+1 ms)Flushing stores of testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. at 1676174330001 (+2 ms)Flushing e019e3f4c00b3da154d59b4085911d76/cf: creating writer at 1676174330001Flushing mob file e019e3f4c00b3da154d59b4085911d76/cf: appending metadata at 1676174330013 (+12 ms)Flushing mob file e019e3f4c00b3da154d59b4085911d76/cf: closing flushed file at 1676174330013Flushing e019e3f4c00b3da154d59b4085911d76/cf: appending metadata at 1676174330034 (+21 ms)Flushing e019e3f4c00b3da154d59b4085911d76/cf: closing flushed file at 1676174330034Flushing e019e3f4c00b3da154d59b4085911d76/cf: reopening flushed file at 1676174330050 (+16 ms)Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for e019e3f4c00b3da154d59b4085911d76 in 57ms, sequenceid=15, compaction requested=false at 1676174330055 (+5 ms)Writing region close event to WAL at 1676174330063 (+8 ms)Running coprocessor post-close hooks at 1676174330067 (+4 ms)Closed at 1676174330068 (+1 ms) 2023-02-12 03:58:50,069 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:50,070 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for cc0e0b01138d4eae5df1e5641b484f45: Waiting for close lock at 1676174329999Running coprocessor pre-close hooks at 1676174329999Disabling compacts and flushes for region at 1676174329999Disabling writes for close at 1676174330000 (+1 ms)Obtaining lock to block concurrent updates at 1676174330000Preparing flush snapshotting stores in cc0e0b01138d4eae5df1e5641b484f45 at 1676174330000Finished memstore snapshotting testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45., syncing WAL and waiting on mvcc, flushsize=dataSize=2008, getHeapSize=4560, getOffHeapSize=0, getCellsCount=30 at 1676174330002 (+2 ms)Flushing stores of testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. at 1676174330002Flushing cc0e0b01138d4eae5df1e5641b484f45/cf: creating writer at 1676174330003 (+1 ms)Flushing mob file cc0e0b01138d4eae5df1e5641b484f45/cf: appending metadata at 1676174330013 (+10 ms)Flushing mob file cc0e0b01138d4eae5df1e5641b484f45/cf: closing flushed file at 1676174330013Flushing cc0e0b01138d4eae5df1e5641b484f45/cf: appending metadata at 1676174330031 (+18 ms)Flushing cc0e0b01138d4eae5df1e5641b484f45/cf: closing flushed file at 1676174330031Flushing cc0e0b01138d4eae5df1e5641b484f45/cf: reopening flushed file at 1676174330047 (+16 ms)Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for cc0e0b01138d4eae5df1e5641b484f45 in 51ms, sequenceid=15, compaction requested=false at 1676174330051 (+4 ms)Writing region close event to WAL at 1676174330062 (+11 ms)Running coprocessor post-close hooks at 1676174330068 (+6 ms)Closed at 1676174330069 (+1 ms) 2023-02-12 03:58:50,070 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:50,070 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for b6894e9d4c505cdcd16894575e6b138a: Waiting for close lock at 1676174329994Running coprocessor pre-close hooks at 1676174329994Disabling compacts and flushes for region at 1676174329994Disabling writes for close at 1676174329996 (+2 ms)Obtaining lock to block concurrent updates at 1676174329996Preparing flush snapshotting stores in b6894e9d4c505cdcd16894575e6b138a at 1676174329996Finished memstore snapshotting testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a., syncing WAL and waiting on mvcc, flushsize=dataSize=22575, getHeapSize=48768, getOffHeapSize=0, getCellsCount=337 at 1676174329998 (+2 ms)Flushing stores of testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. at 1676174329999 (+1 ms)Flushing b6894e9d4c505cdcd16894575e6b138a/cf: creating writer at 1676174330000 (+1 ms)Flushing mob file b6894e9d4c505cdcd16894575e6b138a/cf: appending metadata at 1676174330018 (+18 ms)Flushing mob file b6894e9d4c505cdcd16894575e6b138a/cf: closing flushed file at 1676174330018Flushing b6894e9d4c505cdcd16894575e6b138a/cf: appending metadata at 1676174330034 (+16 ms)Flushing b6894e9d4c505cdcd16894575e6b138a/cf: closing flushed file at 1676174330034Flushing b6894e9d4c505cdcd16894575e6b138a/cf: reopening flushed file at 1676174330050 (+16 ms)Finished flush of dataSize ~22.05 KB/22575, heapSize ~47.63 KB/48768, currentSize=0 B/0 for b6894e9d4c505cdcd16894575e6b138a in 59ms, sequenceid=15, compaction requested=false at 1676174330055 (+5 ms)Writing region close event to WAL at 1676174330062 (+7 ms)Running coprocessor post-close hooks at 1676174330069 (+7 ms)Closed at 1676174330070 (+1 ms) 2023-02-12 03:58:50,071 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:50,071 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:50,072 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing ecda59e01589f4f279b19a5420a1e608, disabling compactions & flushes 2023-02-12 03:58:50,072 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:50,072 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:50,072 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. after waiting 0 ms 2023-02-12 03:58:50,073 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:50,073 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing ecda59e01589f4f279b19a5420a1e608 1/1 column families, dataSize=1.70 KB heapSize=3.91 KB 2023-02-12 03:58:50,073 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=CLOSED 2023-02-12 03:58:50,073 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174330073"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174330073"}]},"ts":"1676174330073"} 2023-02-12 03:58:50,073 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:50,073 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:50,073 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 609b1a331ab507675f39b4bdbe7a382c, disabling compactions & flushes 2023-02-12 03:58:50,073 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:50,075 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:50,075 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. after waiting 0 ms 2023-02-12 03:58:50,075 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:50,076 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 609b1a331ab507675f39b4bdbe7a382c 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-12 03:58:50,076 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=CLOSED 2023-02-12 03:58:50,077 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174330076"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174330076"}]},"ts":"1676174330076"} 2023-02-12 03:58:50,077 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:50,078 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:50,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing fcd2ed887eb9f47c07236428879982c1, disabling compactions & flushes 2023-02-12 03:58:50,079 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:50,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:50,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. after waiting 0 ms 2023-02-12 03:58:50,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:50,079 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing fcd2ed887eb9f47c07236428879982c1 1/1 column families, dataSize=2.94 KB heapSize=6.58 KB 2023-02-12 03:58:50,079 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=CLOSED 2023-02-12 03:58:50,080 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174330079"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174330079"}]},"ts":"1676174330079"} 2023-02-12 03:58:50,083 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=84, resume processing ppid=81 2023-02-12 03:58:50,084 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=84, ppid=81, state=SUCCESS; CloseRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,42205,1676174315413 in 237 msec 2023-02-12 03:58:50,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-12 03:58:50,087 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=88, resume processing ppid=77 2023-02-12 03:58:50,087 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=81, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, UNASSIGN in 248 msec 2023-02-12 03:58:50,087 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=88, ppid=77, state=SUCCESS; CloseRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35783,1676174315451 in 233 msec 2023-02-12 03:58:50,087 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=83, resume processing ppid=82 2023-02-12 03:58:50,088 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=83, ppid=82, state=SUCCESS; CloseRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,35785,1676174315481 in 243 msec 2023-02-12 03:58:50,090 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=77, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, UNASSIGN in 251 msec 2023-02-12 03:58:50,090 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=82, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, UNASSIGN in 252 msec 2023-02-12 03:58:50,110 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 2023-02-12 03:58:50,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 2023-02-12 03:58:50,122 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 2023-02-12 03:58:50,122 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/.tmp/cf/25b90241f56e47b781d04ecdbb8f5bd5 2023-02-12 03:58:50,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/.tmp/cf/2e835d2ef04e4f84a4c85680200dc844 2023-02-12 03:58:50,127 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/.tmp/cf/25b90241f56e47b781d04ecdbb8f5bd5 as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/25b90241f56e47b781d04ecdbb8f5bd5 2023-02-12 03:58:50,129 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/.tmp/cf/2e835d2ef04e4f84a4c85680200dc844 as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/2e835d2ef04e4f84a4c85680200dc844 2023-02-12 03:58:50,132 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/.tmp/cf/1029f052cfe34ddb8506a2f181ab2423 2023-02-12 03:58:50,133 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/25b90241f56e47b781d04ecdbb8f5bd5, entries=26, sequenceid=15, filesize=9.3 K 2023-02-12 03:58:50,135 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for ecda59e01589f4f279b19a5420a1e608 in 62ms, sequenceid=15, compaction requested=false 2023-02-12 03:58:50,143 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/2e835d2ef04e4f84a4c85680200dc844, entries=45, sequenceid=15, filesize=12.5 K 2023-02-12 03:58:50,145 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.94 KB/3013, heapSize ~6.56 KB/6720, currentSize=0 B/0 for fcd2ed887eb9f47c07236428879982c1 in 66ms, sequenceid=15, compaction requested=false 2023-02-12 03:58:50,149 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/.tmp/cf/1029f052cfe34ddb8506a2f181ab2423 as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/1029f052cfe34ddb8506a2f181ab2423 2023-02-12 03:58:50,155 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/1029f052cfe34ddb8506a2f181ab2423, entries=31, sequenceid=15, filesize=10.2 K 2023-02-12 03:58:50,157 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 609b1a331ab507675f39b4bdbe7a382c in 81ms, sequenceid=15, compaction requested=false 2023-02-12 03:58:50,167 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 03:58:50,175 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:50,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for ecda59e01589f4f279b19a5420a1e608: Waiting for close lock at 1676174330071Running coprocessor pre-close hooks at 1676174330071Disabling compacts and flushes for region at 1676174330071Disabling writes for close at 1676174330072 (+1 ms)Obtaining lock to block concurrent updates at 1676174330073 (+1 ms)Preparing flush snapshotting stores in ecda59e01589f4f279b19a5420a1e608 at 1676174330073Finished memstore snapshotting testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608., syncing WAL and waiting on mvcc, flushsize=dataSize=1740, getHeapSize=3984, getOffHeapSize=0, getCellsCount=26 at 1676174330073Flushing stores of testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. at 1676174330073Flushing ecda59e01589f4f279b19a5420a1e608/cf: creating writer at 1676174330075 (+2 ms)Flushing mob file ecda59e01589f4f279b19a5420a1e608/cf: appending metadata at 1676174330085 (+10 ms)Flushing mob file ecda59e01589f4f279b19a5420a1e608/cf: closing flushed file at 1676174330085Flushing ecda59e01589f4f279b19a5420a1e608/cf: appending metadata at 1676174330112 (+27 ms)Flushing ecda59e01589f4f279b19a5420a1e608/cf: closing flushed file at 1676174330112Flushing ecda59e01589f4f279b19a5420a1e608/cf: reopening flushed file at 1676174330128 (+16 ms)Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for ecda59e01589f4f279b19a5420a1e608 in 62ms, sequenceid=15, compaction requested=false at 1676174330135 (+7 ms)Writing region close event to WAL at 1676174330139 (+4 ms)Running coprocessor post-close hooks at 1676174330174 (+35 ms)Closed at 1676174330175 (+1 ms) 2023-02-12 03:58:50,179 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:50,180 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=CLOSED 2023-02-12 03:58:50,180 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174330180"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174330180"}]},"ts":"1676174330180"} 2023-02-12 03:58:50,180 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 03:58:50,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 03:58:50,182 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:50,182 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:50,182 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for fcd2ed887eb9f47c07236428879982c1: Waiting for close lock at 1676174330078Running coprocessor pre-close hooks at 1676174330078Disabling compacts and flushes for region at 1676174330078Disabling writes for close at 1676174330079 (+1 ms)Obtaining lock to block concurrent updates at 1676174330079Preparing flush snapshotting stores in fcd2ed887eb9f47c07236428879982c1 at 1676174330079Finished memstore snapshotting testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1., syncing WAL and waiting on mvcc, flushsize=dataSize=3013, getHeapSize=6720, getOffHeapSize=0, getCellsCount=45 at 1676174330079Flushing stores of testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. at 1676174330080 (+1 ms)Flushing fcd2ed887eb9f47c07236428879982c1/cf: creating writer at 1676174330081 (+1 ms)Flushing mob file fcd2ed887eb9f47c07236428879982c1/cf: appending metadata at 1676174330089 (+8 ms)Flushing mob file fcd2ed887eb9f47c07236428879982c1/cf: closing flushed file at 1676174330089Flushing fcd2ed887eb9f47c07236428879982c1/cf: appending metadata at 1676174330114 (+25 ms)Flushing fcd2ed887eb9f47c07236428879982c1/cf: closing flushed file at 1676174330114Flushing fcd2ed887eb9f47c07236428879982c1/cf: reopening flushed file at 1676174330130 (+16 ms)Finished flush of dataSize ~2.94 KB/3013, heapSize ~6.56 KB/6720, currentSize=0 B/0 for fcd2ed887eb9f47c07236428879982c1 in 66ms, sequenceid=15, compaction requested=false at 1676174330145 (+15 ms)Writing region close event to WAL at 1676174330177 (+32 ms)Running coprocessor post-close hooks at 1676174330181 (+4 ms)Closed at 1676174330182 (+1 ms) 2023-02-12 03:58:50,182 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 609b1a331ab507675f39b4bdbe7a382c: Waiting for close lock at 1676174330073Running coprocessor pre-close hooks at 1676174330073Disabling compacts and flushes for region at 1676174330073Disabling writes for close at 1676174330075 (+2 ms)Obtaining lock to block concurrent updates at 1676174330076 (+1 ms)Preparing flush snapshotting stores in 609b1a331ab507675f39b4bdbe7a382c at 1676174330076Finished memstore snapshotting testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c., syncing WAL and waiting on mvcc, flushsize=dataSize=2075, getHeapSize=4704, getOffHeapSize=0, getCellsCount=31 at 1676174330076Flushing stores of testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. at 1676174330077 (+1 ms)Flushing 609b1a331ab507675f39b4bdbe7a382c/cf: creating writer at 1676174330078 (+1 ms)Flushing mob file 609b1a331ab507675f39b4bdbe7a382c/cf: appending metadata at 1676174330086 (+8 ms)Flushing mob file 609b1a331ab507675f39b4bdbe7a382c/cf: closing flushed file at 1676174330087 (+1 ms)Flushing 609b1a331ab507675f39b4bdbe7a382c/cf: appending metadata at 1676174330123 (+36 ms)Flushing 609b1a331ab507675f39b4bdbe7a382c/cf: closing flushed file at 1676174330123Flushing 609b1a331ab507675f39b4bdbe7a382c/cf: reopening flushed file at 1676174330150 (+27 ms)Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 609b1a331ab507675f39b4bdbe7a382c in 81ms, sequenceid=15, compaction requested=false at 1676174330157 (+7 ms)Writing region close event to WAL at 1676174330174 (+17 ms)Running coprocessor post-close hooks at 1676174330181 (+7 ms)Closed at 1676174330182 (+1 ms) 2023-02-12 03:58:50,184 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:50,184 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:50,185 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=CLOSED 2023-02-12 03:58:50,185 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=87, resume processing ppid=78 2023-02-12 03:58:50,185 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174330185"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174330185"}]},"ts":"1676174330185"} 2023-02-12 03:58:50,185 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=87, ppid=78, state=SUCCESS; CloseRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,42205,1676174315413 in 336 msec 2023-02-12 03:58:50,185 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=CLOSED 2023-02-12 03:58:50,185 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174330185"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174330185"}]},"ts":"1676174330185"} 2023-02-12 03:58:50,186 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=78, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, UNASSIGN in 349 msec 2023-02-12 03:58:50,188 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=86, resume processing ppid=80 2023-02-12 03:58:50,188 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=86, ppid=80, state=SUCCESS; CloseRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451 in 341 msec 2023-02-12 03:58:50,189 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=85, resume processing ppid=79 2023-02-12 03:58:50,189 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=85, ppid=79, state=SUCCESS; CloseRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,35785,1676174315481 in 344 msec 2023-02-12 03:58:50,190 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=80, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, UNASSIGN in 352 msec 2023-02-12 03:58:50,191 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=79, resume processing ppid=76 2023-02-12 03:58:50,191 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=79, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, UNASSIGN in 353 msec 2023-02-12 03:58:50,192 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174330192"}]},"ts":"1676174330192"} 2023-02-12 03:58:50,193 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=DISABLED in hbase:meta 2023-02-12 03:58:50,194 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676174318078 to state=DISABLED 2023-02-12 03:58:50,196 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=76, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676174318078 in 366 msec 2023-02-12 03:58:50,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-12 03:58:50,585 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676174318078, procId: 76 completed 2023-02-12 03:58:50,586 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1580): Client=jenkins//148.251.75.209 snapshot request for:{ ss=snaptb1-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } 2023-02-12 03:58:50,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676174330586 (current time:1676174330586). 2023-02-12 03:58:50,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-12 03:58:50,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] zookeeper.ReadOnlyZKClient(139): Connect 0x41f8b4e9 to 127.0.0.1:50368 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 03:58:50,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d87af0a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 03:58:50,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:50,595 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38204, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:50,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41f8b4e9 to 127.0.0.1:50368 2023-02-12 03:58:50,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:58:50,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-12 03:58:50,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb1-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } 2023-02-12 03:58:50,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE 2023-02-12 03:58:50,599 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE 2023-02-12 03:58:50,600 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=89, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE; timeout=600000, timestamp=1676174930600 2023-02-12 03:58:50,600 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb1-1676174318078 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676174318078 2023-02-12 03:58:50,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb1-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } 2023-02-12 03:58:50,601 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb1-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-12 03:58:50,601 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-12 03:58:50,611 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676174318078 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } 2023-02-12 03:58:50,613 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:50,613 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:50,613 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:50,614 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:50,614 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:50,614 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:50,615 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:50,615 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:50,615 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:50,615 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:50,615 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:50,615 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:50,615 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:50,615 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:50,619 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/027aff06c1054aaaa903715be8404fab, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/10e6966d5cc341688a6163ff0d1d3493] hfiles 2023-02-12 03:58:50,619 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/6ae5fbf6b3ca4a899b9e29e73711608c, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/edd4e077572a45bcac8218359e3d1f91] hfiles 2023-02-12 03:58:50,620 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/25b90241f56e47b781d04ecdbb8f5bd5, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/f75b9c751c584507ac2da36d4b2eb053] hfiles 2023-02-12 03:58:50,619 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/027aff06c1054aaaa903715be8404fab 2023-02-12 03:58:50,620 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/25b90241f56e47b781d04ecdbb8f5bd5 2023-02-12 03:58:50,620 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/6ae5fbf6b3ca4a899b9e29e73711608c 2023-02-12 03:58:50,620 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/541575adc02c4606b85cbfea9970d250, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/6148ef2edb69420e9208fff789b259f0] hfiles 2023-02-12 03:58:50,620 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/f75b9c751c584507ac2da36d4b2eb053 2023-02-12 03:58:50,620 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/10e6966d5cc341688a6163ff0d1d3493 2023-02-12 03:58:50,620 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/541575adc02c4606b85cbfea9970d250 2023-02-12 03:58:50,620 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/edd4e077572a45bcac8218359e3d1f91 2023-02-12 03:58:50,620 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/6148ef2edb69420e9208fff789b259f0 2023-02-12 03:58:50,621 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/1029f052cfe34ddb8506a2f181ab2423, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/de3f19ac4a9247fbac10bbe6eed0d5d1] hfiles 2023-02-12 03:58:50,621 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/1029f052cfe34ddb8506a2f181ab2423 2023-02-12 03:58:50,621 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/de3f19ac4a9247fbac10bbe6eed0d5d1 2023-02-12 03:58:50,621 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/2e835d2ef04e4f84a4c85680200dc844, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/f73ec6d83ce444e98c68cfc9bb0662a3] hfiles 2023-02-12 03:58:50,621 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/2e835d2ef04e4f84a4c85680200dc844 2023-02-12 03:58:50,621 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/f73ec6d83ce444e98c68cfc9bb0662a3 2023-02-12 03:58:50,624 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28] hfiles 2023-02-12 03:58:50,624 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f 2023-02-12 03:58:50,624 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b 2023-02-12 03:58:50,624 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 2023-02-12 03:58:50,624 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 2023-02-12 03:58:50,625 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 2023-02-12 03:58:50,625 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c 2023-02-12 03:58:50,625 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b 2023-02-12 03:58:50,625 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 2023-02-12 03:58:50,625 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 2023-02-12 03:58:50,625 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 2023-02-12 03:58:50,625 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 2023-02-12 03:58:50,625 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 2023-02-12 03:58:50,851 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 03:58:50,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb1-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } is done 2023-02-12 03:58:50,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(404): Snapshoting '{ ss=snaptb1-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 }' is still in progress! 2023-02-12 03:58:50,854 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-12 03:58:51,059 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb1-1676174318078 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } as finished. 2023-02-12 03:58:51,059 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb1-1676174318078 2023-02-12 03:58:51,061 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/snaptb1-1676174318078 2023-02-12 03:58:51,087 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/snaptb1-1676174318078 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/snaptb1-1676174318078 2023-02-12 03:58:51,107 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb1-1676174318078 of table testRestoreSnapshot-1676174318078 completed 2023-02-12 03:58:51,107 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/snaptb1-1676174318078 2023-02-12 03:58:51,107 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/snaptb1-1676174318078 2023-02-12 03:58:51,107 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb1-1676174318078 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676174318078 at 1676174330600Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676174318078 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } at 1676174330612 (+12 ms)Consolidate snapshot: snaptb1-1676174318078 at 1676174331059 (+447 ms)Loading Region manifests for snaptb1-1676174318078 at 1676174331060 (+1 ms)Writing data manifest for snaptb1-1676174318078 at 1676174331067 (+7 ms)Verifying snapshot: snaptb1-1676174318078 at 1676174331076 (+9 ms)Snapshot snaptb1-1676174318078 of table testRestoreSnapshot-1676174318078 completed at 1676174331107 (+31 ms) 2023-02-12 03:58:51,109 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=89, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE 2023-02-12 03:58:51,110 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=89, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE in 511 msec 2023-02-12 03:58:51,354 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 03:58:51,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb1-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } is done 2023-02-12 03:58:51,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb1-1676174318078 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-12 03:58:51,379 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb0-1676174318078-1676174331379 2023-02-12 03:58:51,380 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1580): Client=jenkins//148.251.75.209 snapshot request for:{ ss=hbase-failsafe-snaptb0-1676174318078-1676174331379 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } 2023-02-12 03:58:51,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676174331380 (current time:1676174331380). 2023-02-12 03:58:51,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-12 03:58:51,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] zookeeper.ReadOnlyZKClient(139): Connect 0x63cca028 to 127.0.0.1:50368 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 03:58:51,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fbb124c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 03:58:51,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:51,389 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38210, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:51,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63cca028 to 127.0.0.1:50368 2023-02-12 03:58:51,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:58:51,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-12 03:58:51,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb0-1676174318078-1676174331379 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } 2023-02-12 03:58:51,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE 2023-02-12 03:58:51,393 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE 2023-02-12 03:58:51,394 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=90, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE; timeout=600000, timestamp=1676174931394 2023-02-12 03:58:51,394 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb0-1676174318078-1676174331379 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676174318078 2023-02-12 03:58:51,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb0-1676174318078-1676174331379 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } 2023-02-12 03:58:51,394 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb0-1676174318078-1676174331379 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-12 03:58:51,395 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-12 03:58:51,404 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676174318078-1676174331379 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } 2023-02-12 03:58:51,406 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:51,406 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:51,406 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:51,406 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:51,406 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:51,406 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:51,406 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:51,407 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:51,407 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:51,407 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:51,407 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:51,407 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:51,407 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:51,407 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:51,409 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/1029f052cfe34ddb8506a2f181ab2423, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/de3f19ac4a9247fbac10bbe6eed0d5d1] hfiles 2023-02-12 03:58:51,409 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/541575adc02c4606b85cbfea9970d250, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/6148ef2edb69420e9208fff789b259f0] hfiles 2023-02-12 03:58:51,410 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/1029f052cfe34ddb8506a2f181ab2423 2023-02-12 03:58:51,410 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/541575adc02c4606b85cbfea9970d250 2023-02-12 03:58:51,410 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/de3f19ac4a9247fbac10bbe6eed0d5d1 2023-02-12 03:58:51,410 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/6148ef2edb69420e9208fff789b259f0 2023-02-12 03:58:51,410 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/027aff06c1054aaaa903715be8404fab, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/10e6966d5cc341688a6163ff0d1d3493] hfiles 2023-02-12 03:58:51,410 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/027aff06c1054aaaa903715be8404fab 2023-02-12 03:58:51,410 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/10e6966d5cc341688a6163ff0d1d3493 2023-02-12 03:58:51,417 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/6ae5fbf6b3ca4a899b9e29e73711608c, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/edd4e077572a45bcac8218359e3d1f91] hfiles 2023-02-12 03:58:51,418 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/6ae5fbf6b3ca4a899b9e29e73711608c 2023-02-12 03:58:51,418 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/edd4e077572a45bcac8218359e3d1f91 2023-02-12 03:58:51,418 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/25b90241f56e47b781d04ecdbb8f5bd5, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/f75b9c751c584507ac2da36d4b2eb053] hfiles 2023-02-12 03:58:51,418 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/25b90241f56e47b781d04ecdbb8f5bd5 2023-02-12 03:58:51,418 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/f75b9c751c584507ac2da36d4b2eb053 2023-02-12 03:58:51,420 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28] hfiles 2023-02-12 03:58:51,420 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f 2023-02-12 03:58:51,420 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b 2023-02-12 03:58:51,420 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 2023-02-12 03:58:51,420 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 2023-02-12 03:58:51,420 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 2023-02-12 03:58:51,420 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c 2023-02-12 03:58:51,420 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b 2023-02-12 03:58:51,421 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 2023-02-12 03:58:51,421 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 2023-02-12 03:58:51,421 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 2023-02-12 03:58:51,421 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 2023-02-12 03:58:51,421 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 2023-02-12 03:58:51,421 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/2e835d2ef04e4f84a4c85680200dc844, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/f73ec6d83ce444e98c68cfc9bb0662a3] hfiles 2023-02-12 03:58:51,421 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/2e835d2ef04e4f84a4c85680200dc844 2023-02-12 03:58:51,421 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/f73ec6d83ce444e98c68cfc9bb0662a3 2023-02-12 03:58:51,645 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 03:58:51,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb0-1676174318078-1676174331379 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } is done 2023-02-12 03:58:51,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-snaptb0-1676174318078-1676174331379 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 }' is still in progress! 2023-02-12 03:58:51,646 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-12 03:58:51,842 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb0-1676174318078-1676174331379 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } as finished. 2023-02-12 03:58:51,842 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb0-1676174318078-1676174331379 2023-02-12 03:58:51,843 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676174318078-1676174331379 2023-02-12 03:58:51,874 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676174318078-1676174331379 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/hbase-failsafe-snaptb0-1676174318078-1676174331379 2023-02-12 03:58:51,906 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb0-1676174318078-1676174331379 of table testRestoreSnapshot-1676174318078 completed 2023-02-12 03:58:51,906 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676174318078-1676174331379 2023-02-12 03:58:51,906 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676174318078-1676174331379 2023-02-12 03:58:51,906 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb0-1676174318078-1676174331379 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676174318078 at 1676174331394Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676174318078-1676174331379 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } at 1676174331405 (+11 ms)Consolidate snapshot: hbase-failsafe-snaptb0-1676174318078-1676174331379 at 1676174331842 (+437 ms)Loading Region manifests for hbase-failsafe-snaptb0-1676174318078-1676174331379 at 1676174331842Writing data manifest for hbase-failsafe-snaptb0-1676174318078-1676174331379 at 1676174331849 (+7 ms)Verifying snapshot: hbase-failsafe-snaptb0-1676174318078-1676174331379 at 1676174331862 (+13 ms)Snapshot hbase-failsafe-snaptb0-1676174318078-1676174331379 of table testRestoreSnapshot-1676174318078 completed at 1676174331906 (+44 ms) 2023-02-12 03:58:51,907 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=90, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE 2023-02-12 03:58:51,909 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=90, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE in 515 msec 2023-02-12 03:58:52,146 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 03:58:52,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb0-1676174318078-1676174331379 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } is done 2023-02-12 03:58:52,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb0-1676174318078-1676174331379 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-12 03:58:52,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=91, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676174318078 snapshot=name: "snaptb0-1676174318078" table: "testRestoreSnapshot-1676174318078" creation_time: 1676174323683 type: DISABLED version: 2 ttl: 0 ) 2023-02-12 03:58:52,173 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(880): Restore snapshot=snaptb0-1676174318078 as table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:52,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-12 03:58:52,188 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/.tabledesc/.tableinfo.0000000002 2023-02-12 03:58:52,189 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/.tabledesc/.tableinfo.0000000001 2023-02-12 03:58:52,189 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/.tabledesc/.tableinfo.0000000002 2023-02-12 03:58:52,191 INFO [PEWorker-5] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb0-1676174318078 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } 2023-02-12 03:58:52,196 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb0-1676174318078" table: "testRestoreSnapshot-1676174318078" creation_time: 1676174323683 type: DISABLED version: 2 ttl: 0 2023-02-12 03:58:52,196 DEBUG [PEWorker-5] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078 2023-02-12 03:58:52,209 DEBUG [PEWorker-5] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:52,222 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 2023-02-12 03:58:52,224 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 2023-02-12 03:58:52,226 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 2023-02-12 03:58:52,232 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 2023-02-12 03:58:52,233 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 2023-02-12 03:58:52,235 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f 2023-02-12 03:58:52,235 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:52,235 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:52,235 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:52,235 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:52,235 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:52,235 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:52,242 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/6ae5fbf6b3ca4a899b9e29e73711608c to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/6ae5fbf6b3ca4a899b9e29e73711608c 2023-02-12 03:58:52,243 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/25b90241f56e47b781d04ecdbb8f5bd5 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/25b90241f56e47b781d04ecdbb8f5bd5 2023-02-12 03:58:52,244 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/6148ef2edb69420e9208fff789b259f0 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/6148ef2edb69420e9208fff789b259f0 2023-02-12 03:58:52,244 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/1029f052cfe34ddb8506a2f181ab2423 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/1029f052cfe34ddb8506a2f181ab2423 2023-02-12 03:58:52,244 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/10e6966d5cc341688a6163ff0d1d3493 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/10e6966d5cc341688a6163ff0d1d3493 2023-02-12 03:58:52,244 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/2e835d2ef04e4f84a4c85680200dc844 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/2e835d2ef04e4f84a4c85680200dc844 2023-02-12 03:58:52,244 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb0-1676174318078" table: "testRestoreSnapshot-1676174318078" creation_time: 1676174323683 type: DISABLED version: 2 ttl: 0 2023-02-12 03:58:52,246 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174332245"}]},"ts":"9223372036854775807"} 2023-02-12 03:58:52,246 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174332245"}]},"ts":"9223372036854775807"} 2023-02-12 03:58:52,246 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174332245"}]},"ts":"9223372036854775807"} 2023-02-12 03:58:52,246 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174332245"}]},"ts":"9223372036854775807"} 2023-02-12 03:58:52,246 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174332245"}]},"ts":"9223372036854775807"} 2023-02-12 03:58:52,247 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174332245"}]},"ts":"9223372036854775807"} 2023-02-12 03:58:52,250 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-12 03:58:52,250 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 609b1a331ab507675f39b4bdbe7a382c, NAME => 'testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => b6894e9d4c505cdcd16894575e6b138a, NAME => 'testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => cc0e0b01138d4eae5df1e5641b484f45, NAME => 'testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => e019e3f4c00b3da154d59b4085911d76, NAME => 'testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => ecda59e01589f4f279b19a5420a1e608, NAME => 'testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => fcd2ed887eb9f47c07236428879982c1, NAME => 'testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.', STARTKEY => '2', ENDKEY => '3'}] 2023-02-12 03:58:52,250 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174332246"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174332246"}]},"ts":"1676174332246"} 2023-02-12 03:58:52,250 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174332246"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174332246"}]},"ts":"1676174332246"} 2023-02-12 03:58:52,250 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174332246"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174332246"}]},"ts":"1676174332246"} 2023-02-12 03:58:52,251 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174332246"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174332246"}]},"ts":"1676174332246"} 2023-02-12 03:58:52,251 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174332246"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174332246"}]},"ts":"1676174332246"} 2023-02-12 03:58:52,251 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174332246"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174332246"}]},"ts":"1676174332246"} 2023-02-12 03:58:52,253 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-12 03:58:52,253 INFO [PEWorker-5] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-12 03:58:52,253 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 609b1a331ab507675f39b4bdbe7a382c, NAME => 'testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => b6894e9d4c505cdcd16894575e6b138a, NAME => 'testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => cc0e0b01138d4eae5df1e5641b484f45, NAME => 'testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => e019e3f4c00b3da154d59b4085911d76, NAME => 'testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => ecda59e01589f4f279b19a5420a1e608, NAME => 'testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => fcd2ed887eb9f47c07236428879982c1, NAME => 'testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.', STARTKEY => '2', ENDKEY => '3'}] 2023-02-12 03:58:52,254 INFO [PEWorker-5] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb0-1676174318078 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676174318078 completed! 2023-02-12 03:58:52,257 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=91, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676174318078 snapshot=name: "snaptb0-1676174318078" table: "testRestoreSnapshot-1676174318078" creation_time: 1676174323683 type: DISABLED version: 2 ttl: 0 ) in 91 msec 2023-02-12 03:58:52,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-12 03:58:52,428 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676174318078, procId: 91 completed 2023-02-12 03:58:52,428 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb0-1676174318078-1676174331379 2023-02-12 03:58:52,431 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(748): Client=jenkins//148.251.75.209 delete name: "hbase-failsafe-snaptb0-1676174318078-1676174331379" 2023-02-12 03:58:52,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb0-1676174318078-1676174331379 2023-02-12 03:58:52,436 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676174318078 2023-02-12 03:58:52,436 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$10(2372): Client=jenkins//148.251.75.209 enable testRestoreSnapshot-1676174318078 2023-02-12 03:58:52,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=92, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:52,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-12 03:58:52,439 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676174318078 2023-02-12 03:58:52,439 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174332439"}]},"ts":"1676174332439"} 2023-02-12 03:58:52,441 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=ENABLING in hbase:meta 2023-02-12 03:58:52,443 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-12 03:58:52,445 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN}, {pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN}, {pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN}, {pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN}, {pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN}, {pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN}] 2023-02-12 03:58:52,447 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN 2023-02-12 03:58:52,447 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN 2023-02-12 03:58:52,447 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN 2023-02-12 03:58:52,447 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN 2023-02-12 03:58:52,448 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN 2023-02-12 03:58:52,448 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 03:58:52,448 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 03:58:52,448 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 03:58:52,448 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 03:58:52,449 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 03:58:52,449 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN 2023-02-12 03:58:52,450 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 03:58:52,598 DEBUG [jenkins-hbase20:37305] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-02-12 03:58:52,598 DEBUG [jenkins-hbase20:37305] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-12 03:58:52,598 DEBUG [jenkins-hbase20:37305] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-12 03:58:52,599 DEBUG [jenkins-hbase20:37305] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-12 03:58:52,599 DEBUG [jenkins-hbase20:37305] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-12 03:58:52,601 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:52,601 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:52,601 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:52,602 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174332601"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174332601"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174332601"}]},"ts":"1676174332601"} 2023-02-12 03:58:52,601 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:52,601 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:52,602 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174332601"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174332601"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174332601"}]},"ts":"1676174332601"} 2023-02-12 03:58:52,602 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174332601"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174332601"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174332601"}]},"ts":"1676174332601"} 2023-02-12 03:58:52,602 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174332601"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174332601"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174332601"}]},"ts":"1676174332601"} 2023-02-12 03:58:52,602 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174332601"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174332601"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174332601"}]},"ts":"1676174332601"} 2023-02-12 03:58:52,603 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=99, ppid=98, state=RUNNABLE; OpenRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:52,604 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=100, ppid=96, state=RUNNABLE; OpenRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:52,604 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=101, ppid=95, state=RUNNABLE; OpenRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:52,605 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=102, ppid=93, state=RUNNABLE; OpenRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:52,606 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=103, ppid=97, state=RUNNABLE; OpenRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:52,608 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:52,608 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174332608"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174332608"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174332608"}]},"ts":"1676174332608"} 2023-02-12 03:58:52,610 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=104, ppid=94, state=RUNNABLE; OpenRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:52,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-12 03:58:52,758 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:52,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => fcd2ed887eb9f47c07236428879982c1, NAME => 'testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.', STARTKEY => '2', ENDKEY => '3'} 2023-02-12 03:58:52,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:52,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:52,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:52,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:52,759 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:52,759 INFO [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:52,759 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => cc0e0b01138d4eae5df1e5641b484f45, NAME => 'testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.', STARTKEY => '', ENDKEY => '1'} 2023-02-12 03:58:52,759 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:52,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:52,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:52,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:52,760 DEBUG [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf 2023-02-12 03:58:52,760 DEBUG [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf 2023-02-12 03:58:52,760 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:52,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ecda59e01589f4f279b19a5420a1e608, NAME => 'testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.', STARTKEY => '1', ENDKEY => '2'} 2023-02-12 03:58:52,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:52,760 INFO [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fcd2ed887eb9f47c07236428879982c1 columnFamilyName cf 2023-02-12 03:58:52,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:52,761 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:52,761 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:52,761 INFO [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:52,761 INFO [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:52,762 DEBUG [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf 2023-02-12 03:58:52,762 DEBUG [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf 2023-02-12 03:58:52,762 DEBUG [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf 2023-02-12 03:58:52,762 DEBUG [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf 2023-02-12 03:58:52,762 INFO [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ecda59e01589f4f279b19a5420a1e608 columnFamilyName cf 2023-02-12 03:58:52,762 INFO [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cc0e0b01138d4eae5df1e5641b484f45 columnFamilyName cf 2023-02-12 03:58:52,764 DEBUG [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/f73ec6d83ce444e98c68cfc9bb0662a3 2023-02-12 03:58:52,767 INFO [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] regionserver.HStore(310): Store=fcd2ed887eb9f47c07236428879982c1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:52,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:52,769 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:52,771 DEBUG [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/027aff06c1054aaaa903715be8404fab 2023-02-12 03:58:52,771 INFO [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] regionserver.HStore(310): Store=cc0e0b01138d4eae5df1e5641b484f45/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:52,771 DEBUG [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/f75b9c751c584507ac2da36d4b2eb053 2023-02-12 03:58:52,771 INFO [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] regionserver.HStore(310): Store=ecda59e01589f4f279b19a5420a1e608/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:52,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:52,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:52,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:52,772 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened fcd2ed887eb9f47c07236428879982c1; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69716753, jitterRate=0.03886057436466217}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:52,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:52,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for fcd2ed887eb9f47c07236428879982c1: Running coprocessor pre-open hook at 1676174332758Writing region info on filesystem at 1676174332758Initializing all the Stores at 1676174332759 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174332759Cleaning up temporary data from old regions at 1676174332769 (+10 ms)Cleaning up detritus from prior splits at 1676174332770 (+1 ms)Running coprocessor post-open hooks at 1676174332772 (+2 ms)Region opened successfully at 1676174332772 2023-02-12 03:58:52,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:52,773 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1., pid=101, masterSystemTime=1676174332755 2023-02-12 03:58:52,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:52,775 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:52,775 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:52,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b6894e9d4c505cdcd16894575e6b138a, NAME => 'testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.', STARTKEY => '5', ENDKEY => ''} 2023-02-12 03:58:52,775 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:52,775 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174332775"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174332775"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174332775"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174332775"}]},"ts":"1676174332775"} 2023-02-12 03:58:52,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:52,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:52,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:52,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:52,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:52,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:52,777 INFO [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:52,777 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened ecda59e01589f4f279b19a5420a1e608; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61946433, jitterRate=-0.07692621648311615}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:52,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for ecda59e01589f4f279b19a5420a1e608: Running coprocessor pre-open hook at 1676174332761Writing region info on filesystem at 1676174332761Initializing all the Stores at 1676174332761Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174332761Cleaning up temporary data from old regions at 1676174332773 (+12 ms)Cleaning up detritus from prior splits at 1676174332774 (+1 ms)Running coprocessor post-open hooks at 1676174332777 (+3 ms)Region opened successfully at 1676174332777 2023-02-12 03:58:52,778 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608., pid=104, masterSystemTime=1676174332758 2023-02-12 03:58:52,778 DEBUG [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf 2023-02-12 03:58:52,778 DEBUG [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf 2023-02-12 03:58:52,779 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened cc0e0b01138d4eae5df1e5641b484f45; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70258011, jitterRate=0.046925947070121765}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:52,779 INFO [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b6894e9d4c505cdcd16894575e6b138a columnFamilyName cf 2023-02-12 03:58:52,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for cc0e0b01138d4eae5df1e5641b484f45: Running coprocessor pre-open hook at 1676174332760Writing region info on filesystem at 1676174332760Initializing all the Stores at 1676174332761 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174332761Cleaning up temporary data from old regions at 1676174332773 (+12 ms)Cleaning up detritus from prior splits at 1676174332773Running coprocessor post-open hooks at 1676174332779 (+6 ms)Region opened successfully at 1676174332779 2023-02-12 03:58:52,779 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=101, resume processing ppid=95 2023-02-12 03:58:52,779 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=101, ppid=95, state=SUCCESS; OpenRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,35785,1676174315481 in 173 msec 2023-02-12 03:58:52,779 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45., pid=102, masterSystemTime=1676174332757 2023-02-12 03:58:52,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:52,780 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:52,780 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:52,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e019e3f4c00b3da154d59b4085911d76, NAME => 'testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.', STARTKEY => '4', ENDKEY => '5'} 2023-02-12 03:58:52,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:52,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:52,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:52,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:52,782 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:52,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:52,782 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174332782"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174332782"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174332782"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174332782"}]},"ts":"1676174332782"} 2023-02-12 03:58:52,782 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:52,782 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:52,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 609b1a331ab507675f39b4bdbe7a382c, NAME => 'testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.', STARTKEY => '3', ENDKEY => '4'} 2023-02-12 03:58:52,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:52,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:52,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:52,783 INFO [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:52,783 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=95, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN in 334 msec 2023-02-12 03:58:52,783 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:52,783 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174332783"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174332783"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174332783"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174332783"}]},"ts":"1676174332783"} 2023-02-12 03:58:52,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:52,784 DEBUG [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf 2023-02-12 03:58:52,784 DEBUG [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf 2023-02-12 03:58:52,784 INFO [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e019e3f4c00b3da154d59b4085911d76 columnFamilyName cf 2023-02-12 03:58:52,787 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=104, resume processing ppid=94 2023-02-12 03:58:52,787 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=104, ppid=94, state=SUCCESS; OpenRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,35783,1676174315451 in 174 msec 2023-02-12 03:58:52,788 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=102, resume processing ppid=93 2023-02-12 03:58:52,788 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=102, ppid=93, state=SUCCESS; OpenRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,42205,1676174315413 in 181 msec 2023-02-12 03:58:52,788 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=94, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN in 344 msec 2023-02-12 03:58:52,789 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=93, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN in 345 msec 2023-02-12 03:58:52,795 INFO [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:52,796 DEBUG [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf 2023-02-12 03:58:52,796 DEBUG [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf 2023-02-12 03:58:52,796 INFO [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 609b1a331ab507675f39b4bdbe7a382c columnFamilyName cf 2023-02-12 03:58:52,798 DEBUG [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/541575adc02c4606b85cbfea9970d250 2023-02-12 03:58:52,798 DEBUG [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/edd4e077572a45bcac8218359e3d1f91 2023-02-12 03:58:52,798 INFO [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] regionserver.HStore(310): Store=e019e3f4c00b3da154d59b4085911d76/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:52,798 INFO [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] regionserver.HStore(310): Store=b6894e9d4c505cdcd16894575e6b138a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:52,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:52,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:52,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:52,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:52,801 DEBUG [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/de3f19ac4a9247fbac10bbe6eed0d5d1 2023-02-12 03:58:52,801 INFO [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] regionserver.HStore(310): Store=609b1a331ab507675f39b4bdbe7a382c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:52,802 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:52,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:52,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:52,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:52,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened b6894e9d4c505cdcd16894575e6b138a; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72701905, jitterRate=0.08334280550479889}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:52,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for b6894e9d4c505cdcd16894575e6b138a: Running coprocessor pre-open hook at 1676174332776Writing region info on filesystem at 1676174332776Initializing all the Stores at 1676174332777 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174332777Cleaning up temporary data from old regions at 1676174332801 (+24 ms)Cleaning up detritus from prior splits at 1676174332801Running coprocessor post-open hooks at 1676174332804 (+3 ms)Region opened successfully at 1676174332804 2023-02-12 03:58:52,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened e019e3f4c00b3da154d59b4085911d76; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63565267, jitterRate=-0.05280371010303497}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:52,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for e019e3f4c00b3da154d59b4085911d76: Running coprocessor pre-open hook at 1676174332780Writing region info on filesystem at 1676174332780Initializing all the Stores at 1676174332783 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174332783Cleaning up temporary data from old regions at 1676174332801 (+18 ms)Cleaning up detritus from prior splits at 1676174332801Running coprocessor post-open hooks at 1676174332804 (+3 ms)Region opened successfully at 1676174332804 2023-02-12 03:58:52,805 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a., pid=99, masterSystemTime=1676174332755 2023-02-12 03:58:52,805 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76., pid=103, masterSystemTime=1676174332758 2023-02-12 03:58:52,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:52,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:52,806 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:52,807 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:52,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:52,807 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 609b1a331ab507675f39b4bdbe7a382c; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70023778, jitterRate=0.04343560338020325}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:52,807 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174332807"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174332807"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174332807"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174332807"}]},"ts":"1676174332807"} 2023-02-12 03:58:52,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 609b1a331ab507675f39b4bdbe7a382c: Running coprocessor pre-open hook at 1676174332784Writing region info on filesystem at 1676174332784Initializing all the Stores at 1676174332786 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174332786Cleaning up temporary data from old regions at 1676174332803 (+17 ms)Cleaning up detritus from prior splits at 1676174332804 (+1 ms)Running coprocessor post-open hooks at 1676174332807 (+3 ms)Region opened successfully at 1676174332807 2023-02-12 03:58:52,807 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:52,807 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:52,808 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c., pid=100, masterSystemTime=1676174332757 2023-02-12 03:58:52,808 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174332807"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174332807"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174332807"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174332807"}]},"ts":"1676174332807"} 2023-02-12 03:58:52,809 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:52,809 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:52,810 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:52,810 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174332809"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174332809"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174332809"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174332809"}]},"ts":"1676174332809"} 2023-02-12 03:58:52,811 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=99, resume processing ppid=98 2023-02-12 03:58:52,811 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=99, ppid=98, state=SUCCESS; OpenRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,35785,1676174315481 in 205 msec 2023-02-12 03:58:52,812 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=103, resume processing ppid=97 2023-02-12 03:58:52,812 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=103, ppid=97, state=SUCCESS; OpenRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,35783,1676174315451 in 203 msec 2023-02-12 03:58:52,812 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=98, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN in 366 msec 2023-02-12 03:58:52,813 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=100, resume processing ppid=96 2023-02-12 03:58:52,813 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=97, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN in 367 msec 2023-02-12 03:58:52,813 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=100, ppid=96, state=SUCCESS; OpenRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,42205,1676174315413 in 207 msec 2023-02-12 03:58:52,815 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=96, resume processing ppid=92 2023-02-12 03:58:52,815 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=96, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN in 368 msec 2023-02-12 03:58:52,815 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174332815"}]},"ts":"1676174332815"} 2023-02-12 03:58:52,817 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=ENABLED in hbase:meta 2023-02-12 03:58:52,818 INFO [PEWorker-4] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676174318078' was successfully enabled. 2023-02-12 03:58:52,832 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=92, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676174318078 in 383 msec 2023-02-12 03:58:53,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-12 03:58:53,192 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676174318078, procId: 92 completed 2023-02-12 03:58:53,192 DEBUG [Listener at localhost.localdomain/46483] zookeeper.ReadOnlyZKClient(139): Connect 0x555200b1 to 127.0.0.1:50368 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 03:58:53,198 DEBUG [Listener at localhost.localdomain/46483] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ed71025, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 03:58:53,200 DEBUG [hconnection-0x58c4a1e2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:53,204 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38222, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:53,210 DEBUG [hconnection-0x58c4a1e2-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:53,213 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50838, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:53,327 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-12 03:58:53,434 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36a3b58e to 127.0.0.1:50368 2023-02-12 03:58:53,434 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:58:53,482 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-12 03:58:53,485 DEBUG [hconnection-0x58c4a1e2-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:53,492 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38646, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:53,612 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-12 03:58:53,720 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-12 03:58:53,811 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-12 03:58:54,990 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676174318078 2023-02-12 03:58:54,991 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$11(2405): Client=jenkins//148.251.75.209 disable testRestoreSnapshot-1676174318078 2023-02-12 03:58:54,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=105, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:54,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-12 03:58:54,995 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174334995"}]},"ts":"1676174334995"} 2023-02-12 03:58:54,996 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=DISABLING in hbase:meta 2023-02-12 03:58:54,997 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676174318078 to state=DISABLING 2023-02-12 03:58:54,998 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, UNASSIGN}, {pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, UNASSIGN}, {pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, UNASSIGN}, {pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, UNASSIGN}, {pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, UNASSIGN}, {pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, UNASSIGN}] 2023-02-12 03:58:55,001 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, UNASSIGN 2023-02-12 03:58:55,001 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, UNASSIGN 2023-02-12 03:58:55,001 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, UNASSIGN 2023-02-12 03:58:55,001 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, UNASSIGN 2023-02-12 03:58:55,001 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, UNASSIGN 2023-02-12 03:58:55,002 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:55,002 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:55,002 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:55,002 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:55,002 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174335002"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174335002"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174335002"}]},"ts":"1676174335002"} 2023-02-12 03:58:55,002 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174335002"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174335002"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174335002"}]},"ts":"1676174335002"} 2023-02-12 03:58:55,002 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174335002"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174335002"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174335002"}]},"ts":"1676174335002"} 2023-02-12 03:58:55,002 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:55,002 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174335002"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174335002"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174335002"}]},"ts":"1676174335002"} 2023-02-12 03:58:55,002 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174335002"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174335002"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174335002"}]},"ts":"1676174335002"} 2023-02-12 03:58:55,004 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=112, ppid=110, state=RUNNABLE; CloseRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:55,006 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=107, state=RUNNABLE; CloseRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:55,009 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=114, ppid=111, state=RUNNABLE; CloseRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:55,009 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=109, state=RUNNABLE; CloseRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:55,011 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, UNASSIGN 2023-02-12 03:58:55,011 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=116, ppid=108, state=RUNNABLE; CloseRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:55,012 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:55,012 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174335012"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174335012"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174335012"}]},"ts":"1676174335012"} 2023-02-12 03:58:55,014 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=117, ppid=106, state=RUNNABLE; CloseRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:55,159 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:55,160 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing ecda59e01589f4f279b19a5420a1e608, disabling compactions & flushes 2023-02-12 03:58:55,160 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:55,160 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:55,160 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. after waiting 0 ms 2023-02-12 03:58:55,160 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:55,163 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:55,163 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:55,165 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing b6894e9d4c505cdcd16894575e6b138a, disabling compactions & flushes 2023-02-12 03:58:55,165 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 609b1a331ab507675f39b4bdbe7a382c, disabling compactions & flushes 2023-02-12 03:58:55,165 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:55,165 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:55,166 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:55,166 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. after waiting 0 ms 2023-02-12 03:58:55,166 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:55,166 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:55,166 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. after waiting 0 ms 2023-02-12 03:58:55,166 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:55,192 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-12 03:58:55,193 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:55,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for ecda59e01589f4f279b19a5420a1e608: Waiting for close lock at 1676174335159Running coprocessor pre-close hooks at 1676174335159Disabling compacts and flushes for region at 1676174335159Disabling writes for close at 1676174335160 (+1 ms)Writing region close event to WAL at 1676174335162 (+2 ms)Running coprocessor post-close hooks at 1676174335193 (+31 ms)Closed at 1676174335193 2023-02-12 03:58:55,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:55,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:55,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing e019e3f4c00b3da154d59b4085911d76, disabling compactions & flushes 2023-02-12 03:58:55,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:55,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:55,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. after waiting 0 ms 2023-02-12 03:58:55,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:55,195 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=CLOSED 2023-02-12 03:58:55,195 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174335195"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174335195"}]},"ts":"1676174335195"} 2023-02-12 03:58:55,198 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=113, resume processing ppid=107 2023-02-12 03:58:55,198 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=113, ppid=107, state=SUCCESS; CloseRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,35783,1676174315451 in 192 msec 2023-02-12 03:58:55,200 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=107, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, UNASSIGN in 200 msec 2023-02-12 03:58:55,213 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-12 03:58:55,213 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-12 03:58:55,214 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:55,214 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 609b1a331ab507675f39b4bdbe7a382c: Waiting for close lock at 1676174335165Running coprocessor pre-close hooks at 1676174335165Disabling compacts and flushes for region at 1676174335165Disabling writes for close at 1676174335166 (+1 ms)Writing region close event to WAL at 1676174335173 (+7 ms)Running coprocessor post-close hooks at 1676174335214 (+41 ms)Closed at 1676174335214 2023-02-12 03:58:55,214 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:55,214 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for b6894e9d4c505cdcd16894575e6b138a: Waiting for close lock at 1676174335163Running coprocessor pre-close hooks at 1676174335163Disabling compacts and flushes for region at 1676174335163Disabling writes for close at 1676174335166 (+3 ms)Writing region close event to WAL at 1676174335173 (+7 ms)Running coprocessor post-close hooks at 1676174335214 (+41 ms)Closed at 1676174335214 2023-02-12 03:58:55,216 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:55,216 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=CLOSED 2023-02-12 03:58:55,216 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:55,216 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:55,216 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174335216"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174335216"}]},"ts":"1676174335216"} 2023-02-12 03:58:55,216 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:55,216 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing cc0e0b01138d4eae5df1e5641b484f45, disabling compactions & flushes 2023-02-12 03:58:55,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing fcd2ed887eb9f47c07236428879982c1, disabling compactions & flushes 2023-02-12 03:58:55,217 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:55,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:55,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. after waiting 0 ms 2023-02-12 03:58:55,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:55,217 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:55,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:55,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. after waiting 0 ms 2023-02-12 03:58:55,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:55,217 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=CLOSED 2023-02-12 03:58:55,217 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174335217"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174335217"}]},"ts":"1676174335217"} 2023-02-12 03:58:55,219 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=115, resume processing ppid=109 2023-02-12 03:58:55,220 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=115, ppid=109, state=SUCCESS; CloseRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,42205,1676174315413 in 209 msec 2023-02-12 03:58:55,220 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=114, resume processing ppid=111 2023-02-12 03:58:55,220 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=114, ppid=111, state=SUCCESS; CloseRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,35785,1676174315481 in 209 msec 2023-02-12 03:58:55,221 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=109, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, UNASSIGN in 221 msec 2023-02-12 03:58:55,222 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=111, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, UNASSIGN in 222 msec 2023-02-12 03:58:55,231 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-12 03:58:55,232 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:55,232 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for e019e3f4c00b3da154d59b4085911d76: Waiting for close lock at 1676174335194Running coprocessor pre-close hooks at 1676174335194Disabling compacts and flushes for region at 1676174335194Disabling writes for close at 1676174335194Writing region close event to WAL at 1676174335216 (+22 ms)Running coprocessor post-close hooks at 1676174335232 (+16 ms)Closed at 1676174335232 2023-02-12 03:58:55,234 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:55,234 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=CLOSED 2023-02-12 03:58:55,234 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174335234"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174335234"}]},"ts":"1676174335234"} 2023-02-12 03:58:55,237 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-12 03:58:55,237 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-12 03:58:55,238 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:55,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for cc0e0b01138d4eae5df1e5641b484f45: Waiting for close lock at 1676174335216Running coprocessor pre-close hooks at 1676174335216Disabling compacts and flushes for region at 1676174335216Disabling writes for close at 1676174335217 (+1 ms)Writing region close event to WAL at 1676174335231 (+14 ms)Running coprocessor post-close hooks at 1676174335238 (+7 ms)Closed at 1676174335238 2023-02-12 03:58:55,238 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:55,238 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for fcd2ed887eb9f47c07236428879982c1: Waiting for close lock at 1676174335217Running coprocessor pre-close hooks at 1676174335217Disabling compacts and flushes for region at 1676174335217Disabling writes for close at 1676174335217Writing region close event to WAL at 1676174335231 (+14 ms)Running coprocessor post-close hooks at 1676174335238 (+7 ms)Closed at 1676174335238 2023-02-12 03:58:55,239 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=112, resume processing ppid=110 2023-02-12 03:58:55,239 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=112, ppid=110, state=SUCCESS; CloseRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,35783,1676174315451 in 232 msec 2023-02-12 03:58:55,240 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:55,240 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=CLOSED 2023-02-12 03:58:55,241 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174335240"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174335240"}]},"ts":"1676174335240"} 2023-02-12 03:58:55,241 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:55,241 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=110, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, UNASSIGN in 241 msec 2023-02-12 03:58:55,241 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=CLOSED 2023-02-12 03:58:55,241 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174335241"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174335241"}]},"ts":"1676174335241"} 2023-02-12 03:58:55,245 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=116, resume processing ppid=108 2023-02-12 03:58:55,245 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=116, ppid=108, state=SUCCESS; CloseRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,35785,1676174315481 in 232 msec 2023-02-12 03:58:55,245 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=117, resume processing ppid=106 2023-02-12 03:58:55,245 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=117, ppid=106, state=SUCCESS; CloseRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,42205,1676174315413 in 230 msec 2023-02-12 03:58:55,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-12 03:58:55,246 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=108, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, UNASSIGN in 247 msec 2023-02-12 03:58:55,246 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=106, resume processing ppid=105 2023-02-12 03:58:55,246 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=106, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, UNASSIGN in 247 msec 2023-02-12 03:58:55,247 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174335247"}]},"ts":"1676174335247"} 2023-02-12 03:58:55,248 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=DISABLED in hbase:meta 2023-02-12 03:58:55,249 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676174318078 to state=DISABLED 2023-02-12 03:58:55,252 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=105, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676174318078 in 259 msec 2023-02-12 03:58:55,570 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-12 03:58:55,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-12 03:58:55,747 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676174318078, procId: 105 completed 2023-02-12 03:58:55,755 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676174318078-1676174335755 2023-02-12 03:58:55,756 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1580): Client=jenkins//148.251.75.209 snapshot request for:{ ss=hbase-failsafe-emptySnaptb-1676174318078-1676174335755 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } 2023-02-12 03:58:55,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676174335756 (current time:1676174335756). 2023-02-12 03:58:55,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-12 03:58:55,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] zookeeper.ReadOnlyZKClient(139): Connect 0x06c2d16a to 127.0.0.1:50368 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 03:58:55,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@cb2ccc6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 03:58:55,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:55,764 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40472, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:55,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06c2d16a to 127.0.0.1:50368 2023-02-12 03:58:55,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:58:55,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-12 03:58:55,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-emptySnaptb-1676174318078-1676174335755 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } 2023-02-12 03:58:55,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE 2023-02-12 03:58:55,768 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE 2023-02-12 03:58:55,769 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=118, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE; timeout=600000, timestamp=1676174935769 2023-02-12 03:58:55,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-emptySnaptb-1676174318078-1676174335755 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } 2023-02-12 03:58:55,769 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676174318078-1676174335755 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676174318078 2023-02-12 03:58:55,769 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-emptySnaptb-1676174318078-1676174335755 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-12 03:58:55,769 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-12 03:58:55,778 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676174318078-1676174335755 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } 2023-02-12 03:58:55,780 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:55,780 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:55,780 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:55,780 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:55,780 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:55,780 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:55,780 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:55,780 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:55,780 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:55,780 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:55,780 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:55,780 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:55,780 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:55,780 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:55,781 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/027aff06c1054aaaa903715be8404fab] hfiles 2023-02-12 03:58:55,781 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/027aff06c1054aaaa903715be8404fab 2023-02-12 03:58:55,783 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/f75b9c751c584507ac2da36d4b2eb053] hfiles 2023-02-12 03:58:55,783 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/541575adc02c4606b85cbfea9970d250] hfiles 2023-02-12 03:58:55,783 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/f75b9c751c584507ac2da36d4b2eb053 2023-02-12 03:58:55,783 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/541575adc02c4606b85cbfea9970d250 2023-02-12 03:58:55,784 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/de3f19ac4a9247fbac10bbe6eed0d5d1] hfiles 2023-02-12 03:58:55,784 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/de3f19ac4a9247fbac10bbe6eed0d5d1 2023-02-12 03:58:55,786 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/edd4e077572a45bcac8218359e3d1f91] hfiles 2023-02-12 03:58:55,786 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/edd4e077572a45bcac8218359e3d1f91 2023-02-12 03:58:55,786 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/f73ec6d83ce444e98c68cfc9bb0662a3] hfiles 2023-02-12 03:58:55,786 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/f73ec6d83ce444e98c68cfc9bb0662a3 2023-02-12 03:58:55,787 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28] hfiles 2023-02-12 03:58:55,787 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b 2023-02-12 03:58:55,787 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 2023-02-12 03:58:55,787 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c 2023-02-12 03:58:55,787 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b 2023-02-12 03:58:55,787 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 2023-02-12 03:58:55,787 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 2023-02-12 03:58:55,841 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-emptySnaptb-1676174318078-1676174335755 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } as finished. 2023-02-12 03:58:55,841 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-emptySnaptb-1676174318078-1676174335755 2023-02-12 03:58:55,842 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676174318078-1676174335755 2023-02-12 03:58:55,861 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676174318078-1676174335755 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/hbase-failsafe-emptySnaptb-1676174318078-1676174335755 2023-02-12 03:58:55,882 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-emptySnaptb-1676174318078-1676174335755 of table testRestoreSnapshot-1676174318078 completed 2023-02-12 03:58:55,882 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676174318078-1676174335755 2023-02-12 03:58:55,882 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676174318078-1676174335755 2023-02-12 03:58:55,882 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676174318078-1676174335755 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676174318078 at 1676174335769Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676174318078-1676174335755 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } at 1676174335778 (+9 ms)Consolidate snapshot: hbase-failsafe-emptySnaptb-1676174318078-1676174335755 at 1676174335841 (+63 ms)Loading Region manifests for hbase-failsafe-emptySnaptb-1676174318078-1676174335755 at 1676174335841Writing data manifest for hbase-failsafe-emptySnaptb-1676174318078-1676174335755 at 1676174335844 (+3 ms)Verifying snapshot: hbase-failsafe-emptySnaptb-1676174318078-1676174335755 at 1676174335854 (+10 ms)Snapshot hbase-failsafe-emptySnaptb-1676174318078-1676174335755 of table testRestoreSnapshot-1676174318078 completed at 1676174335882 (+28 ms) 2023-02-12 03:58:55,884 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=118, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE 2023-02-12 03:58:55,885 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=118, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE in 117 msec 2023-02-12 03:58:56,019 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 03:58:56,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-emptySnaptb-1676174318078-1676174335755 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } is done 2023-02-12 03:58:56,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-emptySnaptb-1676174318078-1676174335755 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-12 03:58:56,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=119, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676174318078 snapshot=name: "emptySnaptb-1676174318078" table: "testRestoreSnapshot-1676174318078" creation_time: 1676174320476 type: DISABLED version: 2 ttl: 0 ) 2023-02-12 03:58:56,028 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(880): Restore snapshot=emptySnaptb-1676174318078 as table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:56,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-12 03:58:56,044 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/.tabledesc/.tableinfo.0000000003 2023-02-12 03:58:56,045 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/.tabledesc/.tableinfo.0000000002 2023-02-12 03:58:56,045 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/.tabledesc/.tableinfo.0000000003 2023-02-12 03:58:56,046 INFO [PEWorker-2] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=emptySnaptb-1676174318078 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } 2023-02-12 03:58:56,048 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "emptySnaptb-1676174318078" table: "testRestoreSnapshot-1676174318078" creation_time: 1676174320476 type: DISABLED version: 2 ttl: 0 2023-02-12 03:58:56,048 DEBUG [PEWorker-2] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078 2023-02-12 03:58:56,055 DEBUG [PEWorker-2] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:56,058 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b 2023-02-12 03:58:56,059 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 2023-02-12 03:58:56,060 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c 2023-02-12 03:58:56,062 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b 2023-02-12 03:58:56,063 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 2023-02-12 03:58:56,064 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 2023-02-12 03:58:56,064 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:56,064 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:56,064 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:56,064 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:56,064 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:56,064 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:56,068 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/edd4e077572a45bcac8218359e3d1f91 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/edd4e077572a45bcac8218359e3d1f91 2023-02-12 03:58:56,069 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/de3f19ac4a9247fbac10bbe6eed0d5d1 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/de3f19ac4a9247fbac10bbe6eed0d5d1 2023-02-12 03:58:56,069 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/f75b9c751c584507ac2da36d4b2eb053 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/f75b9c751c584507ac2da36d4b2eb053 2023-02-12 03:58:56,069 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/541575adc02c4606b85cbfea9970d250 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/541575adc02c4606b85cbfea9970d250 2023-02-12 03:58:56,070 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/027aff06c1054aaaa903715be8404fab to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/027aff06c1054aaaa903715be8404fab 2023-02-12 03:58:56,070 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/f73ec6d83ce444e98c68cfc9bb0662a3 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/f73ec6d83ce444e98c68cfc9bb0662a3 2023-02-12 03:58:56,070 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "emptySnaptb-1676174318078" table: "testRestoreSnapshot-1676174318078" creation_time: 1676174320476 type: DISABLED version: 2 ttl: 0 2023-02-12 03:58:56,071 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174336071"}]},"ts":"9223372036854775807"} 2023-02-12 03:58:56,071 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174336071"}]},"ts":"9223372036854775807"} 2023-02-12 03:58:56,071 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174336071"}]},"ts":"9223372036854775807"} 2023-02-12 03:58:56,071 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174336071"}]},"ts":"9223372036854775807"} 2023-02-12 03:58:56,072 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174336071"}]},"ts":"9223372036854775807"} 2023-02-12 03:58:56,072 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174336071"}]},"ts":"9223372036854775807"} 2023-02-12 03:58:56,075 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-12 03:58:56,075 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 609b1a331ab507675f39b4bdbe7a382c, NAME => 'testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => b6894e9d4c505cdcd16894575e6b138a, NAME => 'testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => cc0e0b01138d4eae5df1e5641b484f45, NAME => 'testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => e019e3f4c00b3da154d59b4085911d76, NAME => 'testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => ecda59e01589f4f279b19a5420a1e608, NAME => 'testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => fcd2ed887eb9f47c07236428879982c1, NAME => 'testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.', STARTKEY => '2', ENDKEY => '3'}] 2023-02-12 03:58:56,075 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174336072"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174336072"}]},"ts":"1676174336072"} 2023-02-12 03:58:56,075 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174336072"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174336072"}]},"ts":"1676174336072"} 2023-02-12 03:58:56,075 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174336072"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174336072"}]},"ts":"1676174336072"} 2023-02-12 03:58:56,076 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174336072"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174336072"}]},"ts":"1676174336072"} 2023-02-12 03:58:56,076 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174336072"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174336072"}]},"ts":"1676174336072"} 2023-02-12 03:58:56,076 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174336072"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174336072"}]},"ts":"1676174336072"} 2023-02-12 03:58:56,078 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-12 03:58:56,078 INFO [PEWorker-2] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-12 03:58:56,078 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 609b1a331ab507675f39b4bdbe7a382c, NAME => 'testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => b6894e9d4c505cdcd16894575e6b138a, NAME => 'testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => cc0e0b01138d4eae5df1e5641b484f45, NAME => 'testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => e019e3f4c00b3da154d59b4085911d76, NAME => 'testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => ecda59e01589f4f279b19a5420a1e608, NAME => 'testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => fcd2ed887eb9f47c07236428879982c1, NAME => 'testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.', STARTKEY => '2', ENDKEY => '3'}] 2023-02-12 03:58:56,078 INFO [PEWorker-2] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=emptySnaptb-1676174318078 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676174318078 completed! 2023-02-12 03:58:56,080 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=119, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676174318078 snapshot=name: "emptySnaptb-1676174318078" table: "testRestoreSnapshot-1676174318078" creation_time: 1676174320476 type: DISABLED version: 2 ttl: 0 ) in 52 msec 2023-02-12 03:58:56,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-12 03:58:56,281 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676174318078, procId: 119 completed 2023-02-12 03:58:56,281 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676174318078-1676174335755 2023-02-12 03:58:56,282 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(748): Client=jenkins//148.251.75.209 delete name: "hbase-failsafe-emptySnaptb-1676174318078-1676174335755" 2023-02-12 03:58:56,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-emptySnaptb-1676174318078-1676174335755 2023-02-12 03:58:56,287 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676174318078 2023-02-12 03:58:56,287 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$10(2372): Client=jenkins//148.251.75.209 enable testRestoreSnapshot-1676174318078 2023-02-12 03:58:56,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=120, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:56,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-12 03:58:56,293 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676174318078 2023-02-12 03:58:56,293 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174336293"}]},"ts":"1676174336293"} 2023-02-12 03:58:56,295 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=ENABLING in hbase:meta 2023-02-12 03:58:56,298 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-12 03:58:56,298 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN}, {pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN}, {pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN}, {pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN}, {pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN}, {pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN}] 2023-02-12 03:58:56,300 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN 2023-02-12 03:58:56,300 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN 2023-02-12 03:58:56,300 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN 2023-02-12 03:58:56,300 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN 2023-02-12 03:58:56,301 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN 2023-02-12 03:58:56,301 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 03:58:56,301 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 03:58:56,301 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 03:58:56,301 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 03:58:56,301 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 03:58:56,302 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN 2023-02-12 03:58:56,302 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 03:58:56,451 DEBUG [jenkins-hbase20:37305] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-02-12 03:58:56,452 DEBUG [jenkins-hbase20:37305] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-12 03:58:56,452 DEBUG [jenkins-hbase20:37305] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-12 03:58:56,452 DEBUG [jenkins-hbase20:37305] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-12 03:58:56,452 DEBUG [jenkins-hbase20:37305] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-12 03:58:56,463 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:56,463 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:56,463 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:56,463 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:56,463 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174336463"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174336463"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174336463"}]},"ts":"1676174336463"} 2023-02-12 03:58:56,463 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:56,464 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174336463"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174336463"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174336463"}]},"ts":"1676174336463"} 2023-02-12 03:58:56,463 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174336463"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174336463"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174336463"}]},"ts":"1676174336463"} 2023-02-12 03:58:56,463 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174336463"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174336463"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174336463"}]},"ts":"1676174336463"} 2023-02-12 03:58:56,464 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174336463"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174336463"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174336463"}]},"ts":"1676174336463"} 2023-02-12 03:58:56,465 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=127, ppid=124, state=RUNNABLE; OpenRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:56,465 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=128, ppid=122, state=RUNNABLE; OpenRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:56,466 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=129, ppid=125, state=RUNNABLE; OpenRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:56,466 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=130, ppid=123, state=RUNNABLE; OpenRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:56,467 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=131, ppid=126, state=RUNNABLE; OpenRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:56,473 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:56,473 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174336473"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174336473"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174336473"}]},"ts":"1676174336473"} 2023-02-12 03:58:56,474 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=132, ppid=121, state=RUNNABLE; OpenRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:56,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-12 03:58:56,625 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:56,625 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:56,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b6894e9d4c505cdcd16894575e6b138a, NAME => 'testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.', STARTKEY => '5', ENDKEY => ''} 2023-02-12 03:58:56,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => cc0e0b01138d4eae5df1e5641b484f45, NAME => 'testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.', STARTKEY => '', ENDKEY => '1'} 2023-02-12 03:58:56,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:56,625 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:56,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:56,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ecda59e01589f4f279b19a5420a1e608, NAME => 'testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.', STARTKEY => '1', ENDKEY => '2'} 2023-02-12 03:58:56,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:56,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:56,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:56,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:56,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:56,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:56,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:56,626 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:56,626 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:56,626 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:56,626 INFO [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:56,626 INFO [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:56,626 INFO [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:56,627 DEBUG [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf 2023-02-12 03:58:56,627 DEBUG [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf 2023-02-12 03:58:56,627 DEBUG [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf 2023-02-12 03:58:56,627 DEBUG [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf 2023-02-12 03:58:56,627 DEBUG [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf 2023-02-12 03:58:56,627 DEBUG [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf 2023-02-12 03:58:56,628 INFO [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b6894e9d4c505cdcd16894575e6b138a columnFamilyName cf 2023-02-12 03:58:56,628 INFO [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cc0e0b01138d4eae5df1e5641b484f45 columnFamilyName cf 2023-02-12 03:58:56,628 INFO [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ecda59e01589f4f279b19a5420a1e608 columnFamilyName cf 2023-02-12 03:58:56,629 INFO [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] regionserver.HStore(310): Store=b6894e9d4c505cdcd16894575e6b138a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:56,629 INFO [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] regionserver.HStore(310): Store=cc0e0b01138d4eae5df1e5641b484f45/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:56,629 INFO [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] regionserver.HStore(310): Store=ecda59e01589f4f279b19a5420a1e608/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:56,630 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:56,630 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:56,630 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:56,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:56,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:56,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:56,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:56,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:56,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:56,633 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened b6894e9d4c505cdcd16894575e6b138a; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71274899, jitterRate=0.06207875907421112}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:56,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for b6894e9d4c505cdcd16894575e6b138a: Running coprocessor pre-open hook at 1676174336626Writing region info on filesystem at 1676174336626Initializing all the Stores at 1676174336626Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174336626Cleaning up temporary data from old regions at 1676174336631 (+5 ms)Cleaning up detritus from prior splits at 1676174336631Running coprocessor post-open hooks at 1676174336633 (+2 ms)Region opened successfully at 1676174336633 2023-02-12 03:58:56,633 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened ecda59e01589f4f279b19a5420a1e608; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66390119, jitterRate=-0.010710135102272034}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:56,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for ecda59e01589f4f279b19a5420a1e608: Running coprocessor pre-open hook at 1676174336626Writing region info on filesystem at 1676174336626Initializing all the Stores at 1676174336626Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174336626Cleaning up temporary data from old regions at 1676174336631 (+5 ms)Cleaning up detritus from prior splits at 1676174336631Running coprocessor post-open hooks at 1676174336633 (+2 ms)Region opened successfully at 1676174336633 2023-02-12 03:58:56,633 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened cc0e0b01138d4eae5df1e5641b484f45; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63920566, jitterRate=-0.047509342432022095}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:56,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for cc0e0b01138d4eae5df1e5641b484f45: Running coprocessor pre-open hook at 1676174336626Writing region info on filesystem at 1676174336626Initializing all the Stores at 1676174336626Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174336626Cleaning up temporary data from old regions at 1676174336631 (+5 ms)Cleaning up detritus from prior splits at 1676174336631Running coprocessor post-open hooks at 1676174336633 (+2 ms)Region opened successfully at 1676174336633 2023-02-12 03:58:56,634 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608., pid=128, masterSystemTime=1676174336622 2023-02-12 03:58:56,634 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a., pid=131, masterSystemTime=1676174336622 2023-02-12 03:58:56,634 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45., pid=132, masterSystemTime=1676174336622 2023-02-12 03:58:56,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:56,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:56,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:56,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e019e3f4c00b3da154d59b4085911d76, NAME => 'testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.', STARTKEY => '4', ENDKEY => '5'} 2023-02-12 03:58:56,635 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:56,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:56,635 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174336635"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174336635"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174336635"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174336635"}]},"ts":"1676174336635"} 2023-02-12 03:58:56,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:56,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:56,636 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:56,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:56,636 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:56,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:56,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => fcd2ed887eb9f47c07236428879982c1, NAME => 'testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.', STARTKEY => '2', ENDKEY => '3'} 2023-02-12 03:58:56,636 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:56,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:56,636 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174336636"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174336636"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174336636"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174336636"}]},"ts":"1676174336636"} 2023-02-12 03:58:56,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:56,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:56,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:56,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:56,636 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:56,636 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:56,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 609b1a331ab507675f39b4bdbe7a382c, NAME => 'testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.', STARTKEY => '3', ENDKEY => '4'} 2023-02-12 03:58:56,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:56,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:56,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:56,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:56,637 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:56,637 INFO [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:56,637 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174336637"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174336637"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174336637"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174336637"}]},"ts":"1676174336637"} 2023-02-12 03:58:56,638 INFO [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:56,639 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=128, resume processing ppid=122 2023-02-12 03:58:56,640 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=128, ppid=122, state=SUCCESS; OpenRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,35785,1676174315481 in 172 msec 2023-02-12 03:58:56,640 DEBUG [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf 2023-02-12 03:58:56,640 DEBUG [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf 2023-02-12 03:58:56,640 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=131, resume processing ppid=126 2023-02-12 03:58:56,640 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=131, ppid=126, state=SUCCESS; OpenRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,42205,1676174315413 in 171 msec 2023-02-12 03:58:56,640 DEBUG [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf 2023-02-12 03:58:56,640 DEBUG [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf 2023-02-12 03:58:56,640 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=132, resume processing ppid=121 2023-02-12 03:58:56,641 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=132, ppid=121, state=SUCCESS; OpenRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35783,1676174315451 in 165 msec 2023-02-12 03:58:56,641 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=122, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN in 342 msec 2023-02-12 03:58:56,641 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=126, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN in 342 msec 2023-02-12 03:58:56,641 INFO [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e019e3f4c00b3da154d59b4085911d76 columnFamilyName cf 2023-02-12 03:58:56,641 INFO [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fcd2ed887eb9f47c07236428879982c1 columnFamilyName cf 2023-02-12 03:58:56,641 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=121, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN in 343 msec 2023-02-12 03:58:56,642 INFO [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] regionserver.HStore(310): Store=e019e3f4c00b3da154d59b4085911d76/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:56,642 INFO [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] regionserver.HStore(310): Store=fcd2ed887eb9f47c07236428879982c1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:56,646 INFO [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:56,647 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:56,647 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:56,647 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:56,647 DEBUG [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf 2023-02-12 03:58:56,648 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:56,648 DEBUG [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf 2023-02-12 03:58:56,648 INFO [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 609b1a331ab507675f39b4bdbe7a382c columnFamilyName cf 2023-02-12 03:58:56,649 INFO [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] regionserver.HStore(310): Store=609b1a331ab507675f39b4bdbe7a382c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:56,649 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:56,649 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:56,649 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:56,650 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened e019e3f4c00b3da154d59b4085911d76; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72853144, jitterRate=0.08559644222259521}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:56,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for e019e3f4c00b3da154d59b4085911d76: Running coprocessor pre-open hook at 1676174336636Writing region info on filesystem at 1676174336636Initializing all the Stores at 1676174336637 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174336637Cleaning up temporary data from old regions at 1676174336648 (+11 ms)Cleaning up detritus from prior splits at 1676174336648Running coprocessor post-open hooks at 1676174336650 (+2 ms)Region opened successfully at 1676174336650 2023-02-12 03:58:56,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:56,650 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened fcd2ed887eb9f47c07236428879982c1; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66545335, jitterRate=-0.008397236466407776}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:56,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for fcd2ed887eb9f47c07236428879982c1: Running coprocessor pre-open hook at 1676174336636Writing region info on filesystem at 1676174336636Initializing all the Stores at 1676174336637 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174336637Cleaning up temporary data from old regions at 1676174336648 (+11 ms)Cleaning up detritus from prior splits at 1676174336648Running coprocessor post-open hooks at 1676174336650 (+2 ms)Region opened successfully at 1676174336650 2023-02-12 03:58:56,650 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76., pid=129, masterSystemTime=1676174336622 2023-02-12 03:58:56,651 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1., pid=130, masterSystemTime=1676174336622 2023-02-12 03:58:56,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:56,652 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:56,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:56,653 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:56,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:56,653 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174336652"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174336652"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174336652"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174336652"}]},"ts":"1676174336652"} 2023-02-12 03:58:56,653 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:56,653 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 609b1a331ab507675f39b4bdbe7a382c; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69441092, jitterRate=0.03475290536880493}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:56,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 609b1a331ab507675f39b4bdbe7a382c: Running coprocessor pre-open hook at 1676174336637Writing region info on filesystem at 1676174336637Initializing all the Stores at 1676174336638 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174336638Cleaning up temporary data from old regions at 1676174336650 (+12 ms)Cleaning up detritus from prior splits at 1676174336650Running coprocessor post-open hooks at 1676174336653 (+3 ms)Region opened successfully at 1676174336653 2023-02-12 03:58:56,654 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:56,654 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174336653"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174336653"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174336653"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174336653"}]},"ts":"1676174336653"} 2023-02-12 03:58:56,654 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c., pid=127, masterSystemTime=1676174336622 2023-02-12 03:58:56,659 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:56,659 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:56,660 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:56,660 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=129, resume processing ppid=125 2023-02-12 03:58:56,660 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174336660"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174336660"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174336660"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174336660"}]},"ts":"1676174336660"} 2023-02-12 03:58:56,660 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=129, ppid=125, state=SUCCESS; OpenRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,35785,1676174315481 in 188 msec 2023-02-12 03:58:56,661 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=130, resume processing ppid=123 2023-02-12 03:58:56,661 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=130, ppid=123, state=SUCCESS; OpenRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,42205,1676174315413 in 193 msec 2023-02-12 03:58:56,661 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=125, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN in 362 msec 2023-02-12 03:58:56,662 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=123, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN in 363 msec 2023-02-12 03:58:56,663 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=127, resume processing ppid=124 2023-02-12 03:58:56,663 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=127, ppid=124, state=SUCCESS; OpenRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451 in 197 msec 2023-02-12 03:58:56,664 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=124, resume processing ppid=120 2023-02-12 03:58:56,664 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=124, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN in 365 msec 2023-02-12 03:58:56,664 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174336664"}]},"ts":"1676174336664"} 2023-02-12 03:58:56,665 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=ENABLED in hbase:meta 2023-02-12 03:58:56,667 INFO [PEWorker-4] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676174318078' was successfully enabled. 2023-02-12 03:58:56,668 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=120, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676174318078 in 380 msec 2023-02-12 03:58:57,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-12 03:58:57,045 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676174318078, procId: 120 completed 2023-02-12 03:58:57,045 DEBUG [Listener at localhost.localdomain/46483] zookeeper.ReadOnlyZKClient(139): Connect 0x356c47e3 to 127.0.0.1:50368 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 03:58:57,050 DEBUG [Listener at localhost.localdomain/46483] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@585fafe4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 03:58:57,052 DEBUG [hconnection-0x7c944828-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:57,055 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40474, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:57,059 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-12 03:58:57,062 DEBUG [hconnection-0x7c944828-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:57,065 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50942, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:57,066 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-12 03:58:57,068 DEBUG [hconnection-0x7c944828-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:57,071 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48494, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:57,072 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-12 03:58:57,076 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-12 03:58:57,079 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-12 03:58:57,085 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676174318078 2023-02-12 03:58:57,085 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$11(2405): Client=jenkins//148.251.75.209 disable testRestoreSnapshot-1676174318078 2023-02-12 03:58:57,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=133, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:57,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-12 03:58:57,088 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174337088"}]},"ts":"1676174337088"} 2023-02-12 03:58:57,089 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=DISABLING in hbase:meta 2023-02-12 03:58:57,090 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676174318078 to state=DISABLING 2023-02-12 03:58:57,091 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, UNASSIGN}, {pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, UNASSIGN}, {pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, UNASSIGN}, {pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, UNASSIGN}, {pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, UNASSIGN}, {pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, UNASSIGN}] 2023-02-12 03:58:57,093 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, UNASSIGN 2023-02-12 03:58:57,093 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, UNASSIGN 2023-02-12 03:58:57,093 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, UNASSIGN 2023-02-12 03:58:57,093 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, UNASSIGN 2023-02-12 03:58:57,093 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, UNASSIGN 2023-02-12 03:58:57,094 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:57,094 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:57,094 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174337094"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174337094"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174337094"}]},"ts":"1676174337094"} 2023-02-12 03:58:57,094 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174337094"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174337094"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174337094"}]},"ts":"1676174337094"} 2023-02-12 03:58:57,094 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:57,094 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:57,094 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174337094"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174337094"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174337094"}]},"ts":"1676174337094"} 2023-02-12 03:58:57,094 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174337094"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174337094"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174337094"}]},"ts":"1676174337094"} 2023-02-12 03:58:57,094 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:57,095 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174337094"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174337094"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174337094"}]},"ts":"1676174337094"} 2023-02-12 03:58:57,095 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=140, ppid=137, state=RUNNABLE; CloseRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:57,096 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=141, ppid=136, state=RUNNABLE; CloseRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:57,097 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=142, ppid=138, state=RUNNABLE; CloseRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:57,099 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, UNASSIGN 2023-02-12 03:58:57,099 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=143, ppid=139, state=RUNNABLE; CloseRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:57,099 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=144, ppid=135, state=RUNNABLE; CloseRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:57,099 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:57,099 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174337099"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174337099"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174337099"}]},"ts":"1676174337099"} 2023-02-12 03:58:57,101 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=145, ppid=134, state=RUNNABLE; CloseRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:57,247 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:57,248 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 609b1a331ab507675f39b4bdbe7a382c, disabling compactions & flushes 2023-02-12 03:58:57,248 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:57,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:57,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. after waiting 0 ms 2023-02-12 03:58:57,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:57,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:57,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:57,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing fcd2ed887eb9f47c07236428879982c1, disabling compactions & flushes 2023-02-12 03:58:57,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing ecda59e01589f4f279b19a5420a1e608, disabling compactions & flushes 2023-02-12 03:58:57,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:57,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:57,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:57,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:57,251 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. after waiting 0 ms 2023-02-12 03:58:57,251 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:57,251 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. after waiting 0 ms 2023-02-12 03:58:57,251 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:57,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-12 03:58:57,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-12 03:58:57,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-12 03:58:57,253 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:57,253 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:57,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 609b1a331ab507675f39b4bdbe7a382c: Waiting for close lock at 1676174337248Running coprocessor pre-close hooks at 1676174337248Disabling compacts and flushes for region at 1676174337248Disabling writes for close at 1676174337249 (+1 ms)Writing region close event to WAL at 1676174337249Running coprocessor post-close hooks at 1676174337253 (+4 ms)Closed at 1676174337253 2023-02-12 03:58:57,254 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for fcd2ed887eb9f47c07236428879982c1: Waiting for close lock at 1676174337249Running coprocessor pre-close hooks at 1676174337249Disabling compacts and flushes for region at 1676174337249Disabling writes for close at 1676174337251 (+2 ms)Writing region close event to WAL at 1676174337251Running coprocessor post-close hooks at 1676174337253 (+2 ms)Closed at 1676174337253 2023-02-12 03:58:57,253 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:57,254 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for ecda59e01589f4f279b19a5420a1e608: Waiting for close lock at 1676174337250Running coprocessor pre-close hooks at 1676174337250Disabling compacts and flushes for region at 1676174337250Disabling writes for close at 1676174337251 (+1 ms)Writing region close event to WAL at 1676174337251Running coprocessor post-close hooks at 1676174337253 (+2 ms)Closed at 1676174337253 2023-02-12 03:58:57,255 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:57,255 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:57,256 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing b6894e9d4c505cdcd16894575e6b138a, disabling compactions & flushes 2023-02-12 03:58:57,256 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:57,256 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:57,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. after waiting 0 ms 2023-02-12 03:58:57,257 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=CLOSED 2023-02-12 03:58:57,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:57,257 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174337256"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174337256"}]},"ts":"1676174337256"} 2023-02-12 03:58:57,257 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:57,257 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:57,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing cc0e0b01138d4eae5df1e5641b484f45, disabling compactions & flushes 2023-02-12 03:58:57,258 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:57,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:57,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. after waiting 0 ms 2023-02-12 03:58:57,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:57,258 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=CLOSED 2023-02-12 03:58:57,258 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174337258"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174337258"}]},"ts":"1676174337258"} 2023-02-12 03:58:57,259 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:57,259 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:57,260 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing e019e3f4c00b3da154d59b4085911d76, disabling compactions & flushes 2023-02-12 03:58:57,260 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:57,260 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:57,260 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=CLOSED 2023-02-12 03:58:57,260 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. after waiting 0 ms 2023-02-12 03:58:57,260 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:57,260 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174337260"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174337260"}]},"ts":"1676174337260"} 2023-02-12 03:58:57,262 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-12 03:58:57,263 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:57,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for b6894e9d4c505cdcd16894575e6b138a: Waiting for close lock at 1676174337255Running coprocessor pre-close hooks at 1676174337255Disabling compacts and flushes for region at 1676174337255Disabling writes for close at 1676174337257 (+2 ms)Writing region close event to WAL at 1676174337257Running coprocessor post-close hooks at 1676174337263 (+6 ms)Closed at 1676174337263 2023-02-12 03:58:57,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-12 03:58:57,264 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:57,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for cc0e0b01138d4eae5df1e5641b484f45: Waiting for close lock at 1676174337257Running coprocessor pre-close hooks at 1676174337257Disabling compacts and flushes for region at 1676174337257Disabling writes for close at 1676174337258 (+1 ms)Writing region close event to WAL at 1676174337258Running coprocessor post-close hooks at 1676174337264 (+6 ms)Closed at 1676174337264 2023-02-12 03:58:57,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-12 03:58:57,266 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:57,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for e019e3f4c00b3da154d59b4085911d76: Waiting for close lock at 1676174337259Running coprocessor pre-close hooks at 1676174337259Disabling compacts and flushes for region at 1676174337259Disabling writes for close at 1676174337260 (+1 ms)Writing region close event to WAL at 1676174337261 (+1 ms)Running coprocessor post-close hooks at 1676174337266 (+5 ms)Closed at 1676174337266 2023-02-12 03:58:57,271 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:57,271 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=141, resume processing ppid=136 2023-02-12 03:58:57,271 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=CLOSED 2023-02-12 03:58:57,272 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=141, ppid=136, state=SUCCESS; CloseRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,42205,1676174315413 in 164 msec 2023-02-12 03:58:57,272 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174337271"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174337271"}]},"ts":"1676174337271"} 2023-02-12 03:58:57,272 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:57,272 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=140, resume processing ppid=137 2023-02-12 03:58:57,272 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=140, ppid=137, state=SUCCESS; CloseRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451 in 166 msec 2023-02-12 03:58:57,272 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=CLOSED 2023-02-12 03:58:57,273 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174337272"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174337272"}]},"ts":"1676174337272"} 2023-02-12 03:58:57,274 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:57,274 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=144, resume processing ppid=135 2023-02-12 03:58:57,274 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=136, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, UNASSIGN in 181 msec 2023-02-12 03:58:57,274 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=144, ppid=135, state=SUCCESS; CloseRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,35785,1676174315481 in 172 msec 2023-02-12 03:58:57,274 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=CLOSED 2023-02-12 03:58:57,274 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174337274"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174337274"}]},"ts":"1676174337274"} 2023-02-12 03:58:57,275 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=143, resume processing ppid=139 2023-02-12 03:58:57,275 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=135, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, UNASSIGN in 183 msec 2023-02-12 03:58:57,275 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=143, ppid=139, state=SUCCESS; CloseRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,42205,1676174315413 in 174 msec 2023-02-12 03:58:57,275 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=137, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, UNASSIGN in 183 msec 2023-02-12 03:58:57,276 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=139, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, UNASSIGN in 184 msec 2023-02-12 03:58:57,276 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=145, resume processing ppid=134 2023-02-12 03:58:57,276 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=145, ppid=134, state=SUCCESS; CloseRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35783,1676174315451 in 173 msec 2023-02-12 03:58:57,276 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=134, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, UNASSIGN in 185 msec 2023-02-12 03:58:57,276 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=142, resume processing ppid=138 2023-02-12 03:58:57,277 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=142, ppid=138, state=SUCCESS; CloseRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,35785,1676174315481 in 178 msec 2023-02-12 03:58:57,277 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=138, resume processing ppid=133 2023-02-12 03:58:57,277 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=138, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, UNASSIGN in 186 msec 2023-02-12 03:58:57,278 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174337278"}]},"ts":"1676174337278"} 2023-02-12 03:58:57,279 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=DISABLED in hbase:meta 2023-02-12 03:58:57,280 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676174318078 to state=DISABLED 2023-02-12 03:58:57,281 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=133, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676174318078 in 195 msec 2023-02-12 03:58:57,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-12 03:58:57,340 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676174318078, procId: 133 completed 2023-02-12 03:58:57,349 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb1-1676174318078-1676174337349 2023-02-12 03:58:57,349 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1580): Client=jenkins//148.251.75.209 snapshot request for:{ ss=hbase-failsafe-snaptb1-1676174318078-1676174337349 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } 2023-02-12 03:58:57,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676174337349 (current time:1676174337349). 2023-02-12 03:58:57,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-12 03:58:57,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] zookeeper.ReadOnlyZKClient(139): Connect 0x4e3b5beb to 127.0.0.1:50368 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 03:58:57,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7718a999, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 03:58:57,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:57,357 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40478, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:57,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4e3b5beb to 127.0.0.1:50368 2023-02-12 03:58:57,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:58:57,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-12 03:58:57,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb1-1676174318078-1676174337349 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } 2023-02-12 03:58:57,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE 2023-02-12 03:58:57,361 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE 2023-02-12 03:58:57,362 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=146, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE; timeout=600000, timestamp=1676174937362 2023-02-12 03:58:57,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb1-1676174318078-1676174337349 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } 2023-02-12 03:58:57,362 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb1-1676174318078-1676174337349 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676174318078 2023-02-12 03:58:57,362 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb1-1676174318078-1676174337349 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-12 03:58:57,362 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-12 03:58:57,370 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676174318078-1676174337349 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } 2023-02-12 03:58:57,376 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:57,376 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:57,376 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:57,376 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:57,377 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:57,377 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:57,377 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:57,377 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:57,377 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:57,377 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:57,377 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:57,377 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:57,377 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:58:57,377 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:58:57,378 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 03:58:57,378 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 03:58:57,378 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 03:58:57,378 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 03:58:57,379 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 03:58:57,379 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 03:58:57,380 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 03:58:57,612 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 03:58:57,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1676174318078-1676174337349 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } is done 2023-02-12 03:58:57,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-snaptb1-1676174318078-1676174337349 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 }' is still in progress! 2023-02-12 03:58:57,613 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-12 03:58:57,814 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb1-1676174318078-1676174337349 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } as finished. 2023-02-12 03:58:57,814 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb1-1676174318078-1676174337349 2023-02-12 03:58:57,815 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676174318078-1676174337349 2023-02-12 03:58:57,835 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676174318078-1676174337349 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/hbase-failsafe-snaptb1-1676174318078-1676174337349 2023-02-12 03:58:57,852 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb1-1676174318078-1676174337349 of table testRestoreSnapshot-1676174318078 completed 2023-02-12 03:58:57,852 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676174318078-1676174337349 2023-02-12 03:58:57,852 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676174318078-1676174337349 2023-02-12 03:58:57,852 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb1-1676174318078-1676174337349 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676174318078 at 1676174337362Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676174318078-1676174337349 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } at 1676174337370 (+8 ms)Consolidate snapshot: hbase-failsafe-snaptb1-1676174318078-1676174337349 at 1676174337814 (+444 ms)Loading Region manifests for hbase-failsafe-snaptb1-1676174318078-1676174337349 at 1676174337814Writing data manifest for hbase-failsafe-snaptb1-1676174318078-1676174337349 at 1676174337818 (+4 ms)Verifying snapshot: hbase-failsafe-snaptb1-1676174318078-1676174337349 at 1676174337828 (+10 ms)Snapshot hbase-failsafe-snaptb1-1676174318078-1676174337349 of table testRestoreSnapshot-1676174318078 completed at 1676174337852 (+24 ms) 2023-02-12 03:58:57,853 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=146, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE 2023-02-12 03:58:57,854 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=146, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676174318078, type=EXCLUSIVE in 494 msec 2023-02-12 03:58:58,114 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 03:58:58,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1676174318078-1676174337349 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 } is done 2023-02-12 03:58:58,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb1-1676174318078-1676174337349 table=testRestoreSnapshot-1676174318078 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-12 03:58:58,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=147, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676174318078 snapshot=name: "snaptb1-1676174318078" table: "testRestoreSnapshot-1676174318078" creation_time: 1676174330586 type: DISABLED version: 2 ttl: 0 ) 2023-02-12 03:58:58,131 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(880): Restore snapshot=snaptb1-1676174318078 as table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-12 03:58:58,149 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/.tabledesc/.tableinfo.0000000004 2023-02-12 03:58:58,150 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/.tabledesc/.tableinfo.0000000003 2023-02-12 03:58:58,150 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/.tabledesc/.tableinfo.0000000004 2023-02-12 03:58:58,151 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb1-1676174318078 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } 2023-02-12 03:58:58,153 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676174318078" table: "testRestoreSnapshot-1676174318078" creation_time: 1676174330586 type: DISABLED version: 2 ttl: 0 2023-02-12 03:58:58,153 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,160 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,162 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f of snapshot=snaptb1-1676174318078 to region=c311163dd62fe758baed239605c12349 table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,166 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b of snapshot=snaptb1-1676174318078 to region=c311163dd62fe758baed239605c12349 table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,169 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 of snapshot=snaptb1-1676174318078 to region=c311163dd62fe758baed239605c12349 table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,172 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 of snapshot=snaptb1-1676174318078 to region=c311163dd62fe758baed239605c12349 table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,175 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 of snapshot=snaptb1-1676174318078 to region=c311163dd62fe758baed239605c12349 table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,179 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c of snapshot=snaptb1-1676174318078 to region=c311163dd62fe758baed239605c12349 table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,183 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b of snapshot=snaptb1-1676174318078 to region=c311163dd62fe758baed239605c12349 table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,186 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 of snapshot=snaptb1-1676174318078 to region=c311163dd62fe758baed239605c12349 table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,195 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 of snapshot=snaptb1-1676174318078 to region=c311163dd62fe758baed239605c12349 table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,199 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 of snapshot=snaptb1-1676174318078 to region=c311163dd62fe758baed239605c12349 table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,203 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 of snapshot=snaptb1-1676174318078 to region=c311163dd62fe758baed239605c12349 table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,206 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 of snapshot=snaptb1-1676174318078 to region=c311163dd62fe758baed239605c12349 table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,210 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:58,210 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:58,210 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:58,210 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:58,210 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:58,210 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:58,213 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 1029f052cfe34ddb8506a2f181ab2423 of snapshot=snaptb1-1676174318078 to region=609b1a331ab507675f39b4bdbe7a382c table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,213 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 6ae5fbf6b3ca4a899b9e29e73711608c of snapshot=snaptb1-1676174318078 to region=b6894e9d4c505cdcd16894575e6b138a table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,213 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 027aff06c1054aaaa903715be8404fab of snapshot=snaptb1-1676174318078 to region=cc0e0b01138d4eae5df1e5641b484f45 table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,213 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 541575adc02c4606b85cbfea9970d250 of snapshot=snaptb1-1676174318078 to region=e019e3f4c00b3da154d59b4085911d76 table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,213 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 2e835d2ef04e4f84a4c85680200dc844 of snapshot=snaptb1-1676174318078 to region=fcd2ed887eb9f47c07236428879982c1 table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,214 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 25b90241f56e47b781d04ecdbb8f5bd5 of snapshot=snaptb1-1676174318078 to region=ecda59e01589f4f279b19a5420a1e608 table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,220 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink edd4e077572a45bcac8218359e3d1f91 of snapshot=snaptb1-1676174318078 to region=b6894e9d4c505cdcd16894575e6b138a table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,220 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink f75b9c751c584507ac2da36d4b2eb053 of snapshot=snaptb1-1676174318078 to region=ecda59e01589f4f279b19a5420a1e608 table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,220 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 10e6966d5cc341688a6163ff0d1d3493 of snapshot=snaptb1-1676174318078 to region=cc0e0b01138d4eae5df1e5641b484f45 table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,220 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 6148ef2edb69420e9208fff789b259f0 of snapshot=snaptb1-1676174318078 to region=e019e3f4c00b3da154d59b4085911d76 table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,221 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink f73ec6d83ce444e98c68cfc9bb0662a3 of snapshot=snaptb1-1676174318078 to region=fcd2ed887eb9f47c07236428879982c1 table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,226 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink de3f19ac4a9247fbac10bbe6eed0d5d1 of snapshot=snaptb1-1676174318078 to region=609b1a331ab507675f39b4bdbe7a382c table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,231 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676174318078" table: "testRestoreSnapshot-1676174318078" creation_time: 1676174330586 type: DISABLED version: 2 ttl: 0 2023-02-12 03:58:58,232 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174338232"}]},"ts":"9223372036854775807"} 2023-02-12 03:58:58,232 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174338232"}]},"ts":"9223372036854775807"} 2023-02-12 03:58:58,232 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174338232"}]},"ts":"9223372036854775807"} 2023-02-12 03:58:58,232 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174338232"}]},"ts":"9223372036854775807"} 2023-02-12 03:58:58,232 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174338232"}]},"ts":"9223372036854775807"} 2023-02-12 03:58:58,232 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174338232"}]},"ts":"9223372036854775807"} 2023-02-12 03:58:58,234 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-12 03:58:58,234 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 609b1a331ab507675f39b4bdbe7a382c, NAME => 'testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => b6894e9d4c505cdcd16894575e6b138a, NAME => 'testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => cc0e0b01138d4eae5df1e5641b484f45, NAME => 'testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => e019e3f4c00b3da154d59b4085911d76, NAME => 'testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => ecda59e01589f4f279b19a5420a1e608, NAME => 'testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => fcd2ed887eb9f47c07236428879982c1, NAME => 'testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.', STARTKEY => '2', ENDKEY => '3'}] 2023-02-12 03:58:58,234 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174338233"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174338233"}]},"ts":"1676174338233"} 2023-02-12 03:58:58,234 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174338233"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174338233"}]},"ts":"1676174338233"} 2023-02-12 03:58:58,234 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174338233"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174338233"}]},"ts":"1676174338233"} 2023-02-12 03:58:58,234 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174338233"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174338233"}]},"ts":"1676174338233"} 2023-02-12 03:58:58,234 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174338233"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174338233"}]},"ts":"1676174338233"} 2023-02-12 03:58:58,234 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174338233"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174338233"}]},"ts":"1676174338233"} 2023-02-12 03:58:58,236 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-12 03:58:58,236 INFO [PEWorker-3] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-12 03:58:58,236 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 609b1a331ab507675f39b4bdbe7a382c, NAME => 'testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => b6894e9d4c505cdcd16894575e6b138a, NAME => 'testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => cc0e0b01138d4eae5df1e5641b484f45, NAME => 'testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => e019e3f4c00b3da154d59b4085911d76, NAME => 'testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => ecda59e01589f4f279b19a5420a1e608, NAME => 'testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => fcd2ed887eb9f47c07236428879982c1, NAME => 'testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.', STARTKEY => '2', ENDKEY => '3'}] 2023-02-12 03:58:58,236 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb1-1676174318078 table=testRestoreSnapshot-1676174318078 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676174318078 completed! 2023-02-12 03:58:58,238 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=147, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676174318078 snapshot=name: "snaptb1-1676174318078" table: "testRestoreSnapshot-1676174318078" creation_time: 1676174330586 type: DISABLED version: 2 ttl: 0 ) in 109 msec 2023-02-12 03:58:58,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-12 03:58:58,385 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676174318078, procId: 147 completed 2023-02-12 03:58:58,385 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb1-1676174318078-1676174337349 2023-02-12 03:58:58,386 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(748): Client=jenkins//148.251.75.209 delete name: "hbase-failsafe-snaptb1-1676174318078-1676174337349" 2023-02-12 03:58:58,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb1-1676174318078-1676174337349 2023-02-12 03:58:58,388 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,389 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$10(2372): Client=jenkins//148.251.75.209 enable testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=148, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-12 03:58:58,392 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676174318078 2023-02-12 03:58:58,392 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174338392"}]},"ts":"1676174338392"} 2023-02-12 03:58:58,393 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=ENABLING in hbase:meta 2023-02-12 03:58:58,395 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-12 03:58:58,396 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN}, {pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN}, {pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN}, {pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN}, {pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN}, {pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN}] 2023-02-12 03:58:58,398 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN 2023-02-12 03:58:58,398 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN 2023-02-12 03:58:58,398 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN 2023-02-12 03:58:58,398 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN 2023-02-12 03:58:58,399 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 03:58:58,399 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 03:58:58,399 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 03:58:58,399 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 03:58:58,399 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN 2023-02-12 03:58:58,400 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN 2023-02-12 03:58:58,400 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 03:58:58,401 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 03:58:58,549 DEBUG [jenkins-hbase20:37305] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-02-12 03:58:58,549 DEBUG [jenkins-hbase20:37305] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-12 03:58:58,549 DEBUG [jenkins-hbase20:37305] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-12 03:58:58,549 DEBUG [jenkins-hbase20:37305] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-12 03:58:58,549 DEBUG [jenkins-hbase20:37305] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-12 03:58:58,552 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:58,552 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:58,552 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174338552"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174338552"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174338552"}]},"ts":"1676174338552"} 2023-02-12 03:58:58,552 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174338552"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174338552"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174338552"}]},"ts":"1676174338552"} 2023-02-12 03:58:58,552 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:58,552 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174338552"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174338552"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174338552"}]},"ts":"1676174338552"} 2023-02-12 03:58:58,552 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:58,552 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:58,553 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174338552"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174338552"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174338552"}]},"ts":"1676174338552"} 2023-02-12 03:58:58,553 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174338552"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174338552"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174338552"}]},"ts":"1676174338552"} 2023-02-12 03:58:58,553 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=155, ppid=154, state=RUNNABLE; OpenRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:58,554 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=156, ppid=150, state=RUNNABLE; OpenRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:58,555 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=157, ppid=152, state=RUNNABLE; OpenRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:58,556 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=158, ppid=151, state=RUNNABLE; OpenRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:58:58,559 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:58,559 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174338559"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174338559"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174338559"}]},"ts":"1676174338559"} 2023-02-12 03:58:58,559 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=159, ppid=153, state=RUNNABLE; OpenRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:58:58,560 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=160, ppid=149, state=RUNNABLE; OpenRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:58:58,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-12 03:58:58,709 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:58,709 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:58,709 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e019e3f4c00b3da154d59b4085911d76, NAME => 'testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.', STARTKEY => '4', ENDKEY => '5'} 2023-02-12 03:58:58,709 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => fcd2ed887eb9f47c07236428879982c1, NAME => 'testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.', STARTKEY => '2', ENDKEY => '3'} 2023-02-12 03:58:58,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:58,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:58,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:58,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:58,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:58,710 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:58,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:58,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:58,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:58,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => cc0e0b01138d4eae5df1e5641b484f45, NAME => 'testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.', STARTKEY => '', ENDKEY => '1'} 2023-02-12 03:58:58,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:58,711 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:58,711 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:58,711 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:58,712 INFO [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:58,712 INFO [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:58,712 INFO [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:58,712 DEBUG [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf 2023-02-12 03:58:58,712 DEBUG [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf 2023-02-12 03:58:58,712 DEBUG [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf 2023-02-12 03:58:58,712 DEBUG [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf 2023-02-12 03:58:58,713 DEBUG [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf 2023-02-12 03:58:58,712 DEBUG [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf 2023-02-12 03:58:58,713 INFO [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fcd2ed887eb9f47c07236428879982c1 columnFamilyName cf 2023-02-12 03:58:58,713 INFO [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cc0e0b01138d4eae5df1e5641b484f45 columnFamilyName cf 2023-02-12 03:58:58,713 INFO [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e019e3f4c00b3da154d59b4085911d76 columnFamilyName cf 2023-02-12 03:58:58,732 DEBUG [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/testRestoreSnapshot-1676174318078=fcd2ed887eb9f47c07236428879982c1-2e835d2ef04e4f84a4c85680200dc844 2023-02-12 03:58:58,733 DEBUG [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/testRestoreSnapshot-1676174318078=e019e3f4c00b3da154d59b4085911d76-541575adc02c4606b85cbfea9970d250 2023-02-12 03:58:58,733 DEBUG [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/testRestoreSnapshot-1676174318078=cc0e0b01138d4eae5df1e5641b484f45-027aff06c1054aaaa903715be8404fab 2023-02-12 03:58:58,746 DEBUG [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/testRestoreSnapshot-1676174318078=fcd2ed887eb9f47c07236428879982c1-f73ec6d83ce444e98c68cfc9bb0662a3 2023-02-12 03:58:58,747 INFO [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] regionserver.HStore(310): Store=fcd2ed887eb9f47c07236428879982c1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:58,747 DEBUG [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/testRestoreSnapshot-1676174318078=cc0e0b01138d4eae5df1e5641b484f45-10e6966d5cc341688a6163ff0d1d3493 2023-02-12 03:58:58,747 INFO [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] regionserver.HStore(310): Store=cc0e0b01138d4eae5df1e5641b484f45/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:58,749 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:58,749 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:58,750 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:58,750 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:58,751 DEBUG [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/testRestoreSnapshot-1676174318078=e019e3f4c00b3da154d59b4085911d76-6148ef2edb69420e9208fff789b259f0 2023-02-12 03:58:58,751 INFO [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] regionserver.HStore(310): Store=e019e3f4c00b3da154d59b4085911d76/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:58,751 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:58,752 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:58:58,753 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:58,753 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened fcd2ed887eb9f47c07236428879982c1; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64147471, jitterRate=-0.04412819445133209}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:58,753 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for fcd2ed887eb9f47c07236428879982c1: Running coprocessor pre-open hook at 1676174338710Writing region info on filesystem at 1676174338710Initializing all the Stores at 1676174338711 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174338711Cleaning up temporary data from old regions at 1676174338750 (+39 ms)Cleaning up detritus from prior splits at 1676174338750Running coprocessor post-open hooks at 1676174338753 (+3 ms)Region opened successfully at 1676174338753 2023-02-12 03:58:58,754 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:58:58,754 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1., pid=158, masterSystemTime=1676174338707 2023-02-12 03:58:58,754 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened cc0e0b01138d4eae5df1e5641b484f45; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65049045, jitterRate=-0.030693694949150085}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:58,754 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for cc0e0b01138d4eae5df1e5641b484f45: Running coprocessor pre-open hook at 1676174338711Writing region info on filesystem at 1676174338711Initializing all the Stores at 1676174338711Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174338711Cleaning up temporary data from old regions at 1676174338751 (+40 ms)Cleaning up detritus from prior splits at 1676174338752 (+1 ms)Running coprocessor post-open hooks at 1676174338754 (+2 ms)Region opened successfully at 1676174338754 2023-02-12 03:58:58,755 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45., pid=160, masterSystemTime=1676174338708 2023-02-12 03:58:58,755 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:58:58,756 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:58,756 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:58:58,756 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:58,756 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b6894e9d4c505cdcd16894575e6b138a, NAME => 'testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.', STARTKEY => '5', ENDKEY => ''} 2023-02-12 03:58:58,757 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened e019e3f4c00b3da154d59b4085911d76; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67229789, jitterRate=0.0018019229173660278}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:58,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:58,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for e019e3f4c00b3da154d59b4085911d76: Running coprocessor pre-open hook at 1676174338710Writing region info on filesystem at 1676174338710Initializing all the Stores at 1676174338711 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174338711Cleaning up temporary data from old regions at 1676174338753 (+42 ms)Cleaning up detritus from prior splits at 1676174338753Running coprocessor post-open hooks at 1676174338757 (+4 ms)Region opened successfully at 1676174338757 2023-02-12 03:58:58,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:58,757 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:58,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:58,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:58,757 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174338757"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174338757"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174338757"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174338757"}]},"ts":"1676174338757"} 2023-02-12 03:58:58,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:58,758 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:58:58,758 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:58,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 609b1a331ab507675f39b4bdbe7a382c, NAME => 'testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.', STARTKEY => '3', ENDKEY => '4'} 2023-02-12 03:58:58,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:58,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:58,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:58,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:58,759 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76., pid=159, masterSystemTime=1676174338707 2023-02-12 03:58:58,759 INFO [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:58,760 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:58,760 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174338759"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174338759"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174338759"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174338759"}]},"ts":"1676174338759"} 2023-02-12 03:58:58,760 INFO [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:58,761 DEBUG [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf 2023-02-12 03:58:58,761 DEBUG [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf 2023-02-12 03:58:58,761 INFO [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b6894e9d4c505cdcd16894575e6b138a columnFamilyName cf 2023-02-12 03:58:58,762 DEBUG [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf 2023-02-12 03:58:58,762 DEBUG [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf 2023-02-12 03:58:58,762 INFO [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 609b1a331ab507675f39b4bdbe7a382c columnFamilyName cf 2023-02-12 03:58:58,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:58,763 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:58:58,763 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:58,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ecda59e01589f4f279b19a5420a1e608, NAME => 'testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.', STARTKEY => '1', ENDKEY => '2'} 2023-02-12 03:58:58,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:58,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:58:58,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:58,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:58,764 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:58,764 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174338764"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174338764"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174338764"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174338764"}]},"ts":"1676174338764"} 2023-02-12 03:58:58,764 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=158, resume processing ppid=151 2023-02-12 03:58:58,764 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=158, ppid=151, state=SUCCESS; OpenRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,42205,1676174315413 in 203 msec 2023-02-12 03:58:58,766 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=151, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN in 370 msec 2023-02-12 03:58:58,767 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=160, resume processing ppid=149 2023-02-12 03:58:58,767 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=160, ppid=149, state=SUCCESS; OpenRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35783,1676174315451 in 203 msec 2023-02-12 03:58:58,767 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=159, resume processing ppid=153 2023-02-12 03:58:58,767 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=159, ppid=153, state=SUCCESS; OpenRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,35785,1676174315481 in 207 msec 2023-02-12 03:58:58,768 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=149, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN in 372 msec 2023-02-12 03:58:58,768 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=153, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN in 372 msec 2023-02-12 03:58:58,771 INFO [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:58,772 DEBUG [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf 2023-02-12 03:58:58,772 DEBUG [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf 2023-02-12 03:58:58,772 INFO [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ecda59e01589f4f279b19a5420a1e608 columnFamilyName cf 2023-02-12 03:58:58,783 DEBUG [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/testRestoreSnapshot-1676174318078=ecda59e01589f4f279b19a5420a1e608-25b90241f56e47b781d04ecdbb8f5bd5 2023-02-12 03:58:58,784 DEBUG [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/testRestoreSnapshot-1676174318078=609b1a331ab507675f39b4bdbe7a382c-1029f052cfe34ddb8506a2f181ab2423 2023-02-12 03:58:58,785 DEBUG [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/testRestoreSnapshot-1676174318078=b6894e9d4c505cdcd16894575e6b138a-6ae5fbf6b3ca4a899b9e29e73711608c 2023-02-12 03:58:58,797 DEBUG [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/testRestoreSnapshot-1676174318078=609b1a331ab507675f39b4bdbe7a382c-de3f19ac4a9247fbac10bbe6eed0d5d1 2023-02-12 03:58:58,797 DEBUG [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/testRestoreSnapshot-1676174318078=ecda59e01589f4f279b19a5420a1e608-f75b9c751c584507ac2da36d4b2eb053 2023-02-12 03:58:58,797 INFO [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] regionserver.HStore(310): Store=ecda59e01589f4f279b19a5420a1e608/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:58,797 INFO [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] regionserver.HStore(310): Store=609b1a331ab507675f39b4bdbe7a382c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:58,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:58,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:58,799 DEBUG [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/testRestoreSnapshot-1676174318078=b6894e9d4c505cdcd16894575e6b138a-edd4e077572a45bcac8218359e3d1f91 2023-02-12 03:58:58,799 INFO [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] regionserver.HStore(310): Store=b6894e9d4c505cdcd16894575e6b138a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:58:58,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:58,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:58,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:58,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:58,802 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:58:58,803 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened ecda59e01589f4f279b19a5420a1e608; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73657185, jitterRate=0.09757758677005768}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:58,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for ecda59e01589f4f279b19a5420a1e608: Running coprocessor pre-open hook at 1676174338764Writing region info on filesystem at 1676174338764Initializing all the Stores at 1676174338764Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174338764Cleaning up temporary data from old regions at 1676174338800 (+36 ms)Cleaning up detritus from prior splits at 1676174338800Running coprocessor post-open hooks at 1676174338803 (+3 ms)Region opened successfully at 1676174338803 2023-02-12 03:58:58,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:58:58,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 609b1a331ab507675f39b4bdbe7a382c; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64242437, jitterRate=-0.042713090777397156}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:58,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 609b1a331ab507675f39b4bdbe7a382c: Running coprocessor pre-open hook at 1676174338758Writing region info on filesystem at 1676174338758Initializing all the Stores at 1676174338759 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174338759Cleaning up temporary data from old regions at 1676174338801 (+42 ms)Cleaning up detritus from prior splits at 1676174338802 (+1 ms)Running coprocessor post-open hooks at 1676174338804 (+2 ms)Region opened successfully at 1676174338804 2023-02-12 03:58:58,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:58:58,805 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608., pid=156, masterSystemTime=1676174338707 2023-02-12 03:58:58,805 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c., pid=157, masterSystemTime=1676174338708 2023-02-12 03:58:58,806 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened b6894e9d4c505cdcd16894575e6b138a; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61005759, jitterRate=-0.0909433513879776}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:58:58,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for b6894e9d4c505cdcd16894575e6b138a: Running coprocessor pre-open hook at 1676174338757Writing region info on filesystem at 1676174338757Initializing all the Stores at 1676174338759 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174338759Cleaning up temporary data from old regions at 1676174338801 (+42 ms)Cleaning up detritus from prior splits at 1676174338801Running coprocessor post-open hooks at 1676174338806 (+5 ms)Region opened successfully at 1676174338806 2023-02-12 03:58:58,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:58,807 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:58:58,807 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a., pid=155, masterSystemTime=1676174338707 2023-02-12 03:58:58,807 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:58:58,807 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174338807"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174338807"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174338807"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174338807"}]},"ts":"1676174338807"} 2023-02-12 03:58:58,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:58,808 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:58:58,808 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:58:58,808 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174338808"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174338808"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174338808"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174338808"}]},"ts":"1676174338808"} 2023-02-12 03:58:58,812 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:58:58,812 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174338812"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174338812"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174338812"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174338812"}]},"ts":"1676174338812"} 2023-02-12 03:58:58,814 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=156, resume processing ppid=150 2023-02-12 03:58:58,814 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=156, ppid=150, state=SUCCESS; OpenRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,35785,1676174315481 in 257 msec 2023-02-12 03:58:58,815 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=157, resume processing ppid=152 2023-02-12 03:58:58,815 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=157, ppid=152, state=SUCCESS; OpenRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451 in 258 msec 2023-02-12 03:58:58,816 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=155, resume processing ppid=154 2023-02-12 03:58:58,816 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=150, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN in 419 msec 2023-02-12 03:58:58,816 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=155, ppid=154, state=SUCCESS; OpenRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,42205,1676174315413 in 261 msec 2023-02-12 03:58:58,817 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=152, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN in 420 msec 2023-02-12 03:58:58,817 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:58,817 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:58:58,821 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=154, resume processing ppid=148 2023-02-12 03:58:58,821 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=154, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN in 421 msec 2023-02-12 03:58:58,831 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174338830"}]},"ts":"1676174338830"} 2023-02-12 03:58:58,832 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=ENABLED in hbase:meta 2023-02-12 03:58:58,833 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676174318078' was successfully enabled. 2023-02-12 03:58:58,835 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=148, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676174318078 in 445 msec 2023-02-12 03:58:59,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-12 03:58:59,144 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676174318078, procId: 148 completed 2023-02-12 03:58:59,144 DEBUG [Listener at localhost.localdomain/46483] zookeeper.ReadOnlyZKClient(139): Connect 0x27ddad26 to 127.0.0.1:50368 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 03:58:59,148 DEBUG [Listener at localhost.localdomain/46483] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@452ab791, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 03:58:59,151 DEBUG [hconnection-0x28933828-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:59,154 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40490, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:59,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,392 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-12 03:58:59,396 DEBUG [hconnection-0x28933828-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:59,402 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50958, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:59,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,622 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-12 03:58:59,627 DEBUG [hconnection-0x28933828-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:58:59,630 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48496, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:58:59,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,947 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-12 03:58:59,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:58:59,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,196 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x356c47e3 to 127.0.0.1:50368 2023-02-12 03:59:00,196 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:59:00,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,197 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x555200b1 to 127.0.0.1:50368 2023-02-12 03:59:00,197 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:59:00,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,222 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-12 03:59:00,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,439 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-12 03:59:00,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,697 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-12 03:59:00,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:00,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:01,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:02,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:03,434 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676174318078 2023-02-12 03:59:03,435 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$11(2405): Client=jenkins//148.251.75.209 disable testRestoreSnapshot-1676174318078 2023-02-12 03:59:03,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=161, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:03,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-12 03:59:03,439 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174343438"}]},"ts":"1676174343438"} 2023-02-12 03:59:03,440 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=DISABLING in hbase:meta 2023-02-12 03:59:03,442 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676174318078 to state=DISABLING 2023-02-12 03:59:03,443 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, UNASSIGN}, {pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, UNASSIGN}, {pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, UNASSIGN}, {pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, UNASSIGN}, {pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, UNASSIGN}, {pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, UNASSIGN}] 2023-02-12 03:59:03,444 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, UNASSIGN 2023-02-12 03:59:03,445 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, UNASSIGN 2023-02-12 03:59:03,445 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, UNASSIGN 2023-02-12 03:59:03,445 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, UNASSIGN 2023-02-12 03:59:03,445 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, UNASSIGN 2023-02-12 03:59:03,445 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:03,445 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174343445"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174343445"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174343445"}]},"ts":"1676174343445"} 2023-02-12 03:59:03,446 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:03,446 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174343446"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174343446"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174343446"}]},"ts":"1676174343446"} 2023-02-12 03:59:03,446 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:03,446 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174343445"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174343445"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174343445"}]},"ts":"1676174343445"} 2023-02-12 03:59:03,446 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:03,446 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174343446"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174343446"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174343446"}]},"ts":"1676174343446"} 2023-02-12 03:59:03,446 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:03,446 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174343446"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174343446"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174343446"}]},"ts":"1676174343446"} 2023-02-12 03:59:03,447 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=168, ppid=165, state=RUNNABLE; CloseRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:59:03,447 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=169, ppid=162, state=RUNNABLE; CloseRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:59:03,448 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=170, ppid=163, state=RUNNABLE; CloseRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:59:03,450 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=171, ppid=167, state=RUNNABLE; CloseRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:59:03,451 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=172, ppid=164, state=RUNNABLE; CloseRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:59:03,452 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, UNASSIGN 2023-02-12 03:59:03,453 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:03,453 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174343453"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174343453"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174343453"}]},"ts":"1676174343453"} 2023-02-12 03:59:03,454 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=173, ppid=166, state=RUNNABLE; CloseRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:59:03,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:59:03,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 609b1a331ab507675f39b4bdbe7a382c, disabling compactions & flushes 2023-02-12 03:59:03,603 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:59:03,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:59:03,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. after waiting 0 ms 2023-02-12 03:59:03,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:59:03,604 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:59:03,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing b6894e9d4c505cdcd16894575e6b138a, disabling compactions & flushes 2023-02-12 03:59:03,605 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:59:03,605 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:59:03,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:59:03,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. after waiting 0 ms 2023-02-12 03:59:03,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing ecda59e01589f4f279b19a5420a1e608, disabling compactions & flushes 2023-02-12 03:59:03,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:59:03,607 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:59:03,608 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:59:03,608 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. after waiting 0 ms 2023-02-12 03:59:03,609 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:59:03,618 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-12 03:59:03,618 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-12 03:59:03,618 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-12 03:59:03,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:59:03,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:59:03,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:59:03,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for b6894e9d4c505cdcd16894575e6b138a: Waiting for close lock at 1676174343604Running coprocessor pre-close hooks at 1676174343604Disabling compacts and flushes for region at 1676174343604Disabling writes for close at 1676174343605 (+1 ms)Writing region close event to WAL at 1676174343613 (+8 ms)Running coprocessor post-close hooks at 1676174343619 (+6 ms)Closed at 1676174343620 (+1 ms) 2023-02-12 03:59:03,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 609b1a331ab507675f39b4bdbe7a382c: Waiting for close lock at 1676174343602Running coprocessor pre-close hooks at 1676174343602Disabling compacts and flushes for region at 1676174343602Disabling writes for close at 1676174343604 (+2 ms)Writing region close event to WAL at 1676174343610 (+6 ms)Running coprocessor post-close hooks at 1676174343618 (+8 ms)Closed at 1676174343620 (+2 ms) 2023-02-12 03:59:03,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for ecda59e01589f4f279b19a5420a1e608: Waiting for close lock at 1676174343605Running coprocessor pre-close hooks at 1676174343605Disabling compacts and flushes for region at 1676174343605Disabling writes for close at 1676174343609 (+4 ms)Writing region close event to WAL at 1676174343616 (+7 ms)Running coprocessor post-close hooks at 1676174343619 (+3 ms)Closed at 1676174343620 (+1 ms) 2023-02-12 03:59:03,621 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:59:03,621 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:59:03,623 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing cc0e0b01138d4eae5df1e5641b484f45, disabling compactions & flushes 2023-02-12 03:59:03,623 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:59:03,623 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:59:03,623 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. after waiting 0 ms 2023-02-12 03:59:03,623 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:59:03,623 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=CLOSED 2023-02-12 03:59:03,623 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174343623"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174343623"}]},"ts":"1676174343623"} 2023-02-12 03:59:03,623 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:59:03,623 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:59:03,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing e019e3f4c00b3da154d59b4085911d76, disabling compactions & flushes 2023-02-12 03:59:03,626 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:59:03,626 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=CLOSED 2023-02-12 03:59:03,626 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:59:03,626 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. after waiting 0 ms 2023-02-12 03:59:03,626 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174343626"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174343626"}]},"ts":"1676174343626"} 2023-02-12 03:59:03,626 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:59:03,627 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:59:03,627 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:59:03,628 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing fcd2ed887eb9f47c07236428879982c1, disabling compactions & flushes 2023-02-12 03:59:03,628 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:59:03,628 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:59:03,628 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=CLOSED 2023-02-12 03:59:03,628 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. after waiting 0 ms 2023-02-12 03:59:03,628 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:59:03,628 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174343628"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174343628"}]},"ts":"1676174343628"} 2023-02-12 03:59:03,630 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=168, resume processing ppid=165 2023-02-12 03:59:03,630 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=168, ppid=165, state=SUCCESS; CloseRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451 in 182 msec 2023-02-12 03:59:03,634 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=170, resume processing ppid=163 2023-02-12 03:59:03,634 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=170, ppid=163, state=SUCCESS; CloseRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,35785,1676174315481 in 181 msec 2023-02-12 03:59:03,641 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=171, resume processing ppid=167 2023-02-12 03:59:03,641 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=171, ppid=167, state=SUCCESS; CloseRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,42205,1676174315413 in 180 msec 2023-02-12 03:59:03,641 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=165, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, UNASSIGN in 188 msec 2023-02-12 03:59:03,641 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=163, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, UNASSIGN in 192 msec 2023-02-12 03:59:03,642 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=167, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, UNASSIGN in 199 msec 2023-02-12 03:59:03,642 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-12 03:59:03,643 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:59:03,643 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for cc0e0b01138d4eae5df1e5641b484f45: Waiting for close lock at 1676174343621Running coprocessor pre-close hooks at 1676174343621Disabling compacts and flushes for region at 1676174343621Disabling writes for close at 1676174343623 (+2 ms)Writing region close event to WAL at 1676174343628 (+5 ms)Running coprocessor post-close hooks at 1676174343642 (+14 ms)Closed at 1676174343643 (+1 ms) 2023-02-12 03:59:03,644 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-12 03:59:03,645 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:59:03,646 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:59:03,646 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for fcd2ed887eb9f47c07236428879982c1: Waiting for close lock at 1676174343627Running coprocessor pre-close hooks at 1676174343627Disabling compacts and flushes for region at 1676174343627Disabling writes for close at 1676174343628 (+1 ms)Writing region close event to WAL at 1676174343640 (+12 ms)Running coprocessor post-close hooks at 1676174343645 (+5 ms)Closed at 1676174343646 (+1 ms) 2023-02-12 03:59:03,646 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=CLOSED 2023-02-12 03:59:03,647 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174343646"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174343646"}]},"ts":"1676174343646"} 2023-02-12 03:59:03,647 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-12 03:59:03,649 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:59:03,649 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:59:03,649 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for e019e3f4c00b3da154d59b4085911d76: Waiting for close lock at 1676174343623Running coprocessor pre-close hooks at 1676174343623Disabling compacts and flushes for region at 1676174343623Disabling writes for close at 1676174343626 (+3 ms)Writing region close event to WAL at 1676174343634 (+8 ms)Running coprocessor post-close hooks at 1676174343648 (+14 ms)Closed at 1676174343649 (+1 ms) 2023-02-12 03:59:03,651 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=CLOSED 2023-02-12 03:59:03,651 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174343651"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174343651"}]},"ts":"1676174343651"} 2023-02-12 03:59:03,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:59:03,655 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=169, resume processing ppid=162 2023-02-12 03:59:03,655 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=CLOSED 2023-02-12 03:59:03,655 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174343655"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174343655"}]},"ts":"1676174343655"} 2023-02-12 03:59:03,655 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=169, ppid=162, state=SUCCESS; CloseRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35783,1676174315451 in 203 msec 2023-02-12 03:59:03,656 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=172, resume processing ppid=164 2023-02-12 03:59:03,656 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=172, ppid=164, state=SUCCESS; CloseRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,42205,1676174315413 in 201 msec 2023-02-12 03:59:03,657 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=162, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, UNASSIGN in 213 msec 2023-02-12 03:59:03,658 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=164, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, UNASSIGN in 214 msec 2023-02-12 03:59:03,659 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=173, resume processing ppid=166 2023-02-12 03:59:03,659 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=173, ppid=166, state=SUCCESS; CloseRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,35785,1676174315481 in 203 msec 2023-02-12 03:59:03,660 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=166, resume processing ppid=161 2023-02-12 03:59:03,660 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=166, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, UNASSIGN in 217 msec 2023-02-12 03:59:03,660 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174343660"}]},"ts":"1676174343660"} 2023-02-12 03:59:03,661 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=DISABLED in hbase:meta 2023-02-12 03:59:03,662 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676174318078 to state=DISABLED 2023-02-12 03:59:03,668 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=161, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676174318078 in 232 msec 2023-02-12 03:59:03,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-12 03:59:03,690 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676174318078, procId: 161 completed 2023-02-12 03:59:03,696 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$5(2215): Client=jenkins//148.251.75.209 delete testRestoreSnapshot-1676174318078 2023-02-12 03:59:03,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:03,706 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:03,708 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=174, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:03,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-12 03:59:03,716 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:59:03,716 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:59:03,716 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:59:03,716 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:59:03,716 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:59:03,716 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:59:03,720 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf, FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/recovered.edits] 2023-02-12 03:59:03,720 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf, FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/recovered.edits] 2023-02-12 03:59:03,720 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf, FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/recovered.edits] 2023-02-12 03:59:03,720 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf, FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/recovered.edits] 2023-02-12 03:59:03,721 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf, FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/recovered.edits] 2023-02-12 03:59:03,721 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf, FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/recovered.edits] 2023-02-12 03:59:03,724 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/testRestoreSnapshot-1676174318078=b6894e9d4c505cdcd16894575e6b138a-6ae5fbf6b3ca4a899b9e29e73711608c to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/testRestoreSnapshot-1676174318078=b6894e9d4c505cdcd16894575e6b138a-6ae5fbf6b3ca4a899b9e29e73711608c 2023-02-12 03:59:03,724 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/testRestoreSnapshot-1676174318078=609b1a331ab507675f39b4bdbe7a382c-1029f052cfe34ddb8506a2f181ab2423 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/testRestoreSnapshot-1676174318078=609b1a331ab507675f39b4bdbe7a382c-1029f052cfe34ddb8506a2f181ab2423 2023-02-12 03:59:03,724 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/testRestoreSnapshot-1676174318078=e019e3f4c00b3da154d59b4085911d76-541575adc02c4606b85cbfea9970d250 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/testRestoreSnapshot-1676174318078=e019e3f4c00b3da154d59b4085911d76-541575adc02c4606b85cbfea9970d250 2023-02-12 03:59:03,725 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/testRestoreSnapshot-1676174318078=fcd2ed887eb9f47c07236428879982c1-2e835d2ef04e4f84a4c85680200dc844 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/testRestoreSnapshot-1676174318078=fcd2ed887eb9f47c07236428879982c1-2e835d2ef04e4f84a4c85680200dc844 2023-02-12 03:59:03,726 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/testRestoreSnapshot-1676174318078=ecda59e01589f4f279b19a5420a1e608-25b90241f56e47b781d04ecdbb8f5bd5 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/testRestoreSnapshot-1676174318078=ecda59e01589f4f279b19a5420a1e608-25b90241f56e47b781d04ecdbb8f5bd5 2023-02-12 03:59:03,728 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/testRestoreSnapshot-1676174318078=cc0e0b01138d4eae5df1e5641b484f45-027aff06c1054aaaa903715be8404fab to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/testRestoreSnapshot-1676174318078=cc0e0b01138d4eae5df1e5641b484f45-027aff06c1054aaaa903715be8404fab 2023-02-12 03:59:03,729 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/testRestoreSnapshot-1676174318078=609b1a331ab507675f39b4bdbe7a382c-de3f19ac4a9247fbac10bbe6eed0d5d1 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/testRestoreSnapshot-1676174318078=609b1a331ab507675f39b4bdbe7a382c-de3f19ac4a9247fbac10bbe6eed0d5d1 2023-02-12 03:59:03,729 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/testRestoreSnapshot-1676174318078=b6894e9d4c505cdcd16894575e6b138a-edd4e077572a45bcac8218359e3d1f91 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/testRestoreSnapshot-1676174318078=b6894e9d4c505cdcd16894575e6b138a-edd4e077572a45bcac8218359e3d1f91 2023-02-12 03:59:03,729 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/testRestoreSnapshot-1676174318078=fcd2ed887eb9f47c07236428879982c1-f73ec6d83ce444e98c68cfc9bb0662a3 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/testRestoreSnapshot-1676174318078=fcd2ed887eb9f47c07236428879982c1-f73ec6d83ce444e98c68cfc9bb0662a3 2023-02-12 03:59:03,729 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/testRestoreSnapshot-1676174318078=e019e3f4c00b3da154d59b4085911d76-6148ef2edb69420e9208fff789b259f0 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/testRestoreSnapshot-1676174318078=e019e3f4c00b3da154d59b4085911d76-6148ef2edb69420e9208fff789b259f0 2023-02-12 03:59:03,729 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/testRestoreSnapshot-1676174318078=ecda59e01589f4f279b19a5420a1e608-f75b9c751c584507ac2da36d4b2eb053 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/testRestoreSnapshot-1676174318078=ecda59e01589f4f279b19a5420a1e608-f75b9c751c584507ac2da36d4b2eb053 2023-02-12 03:59:03,737 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/testRestoreSnapshot-1676174318078=cc0e0b01138d4eae5df1e5641b484f45-10e6966d5cc341688a6163ff0d1d3493 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/testRestoreSnapshot-1676174318078=cc0e0b01138d4eae5df1e5641b484f45-10e6966d5cc341688a6163ff0d1d3493 2023-02-12 03:59:03,738 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/recovered.edits/27.seqid to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/recovered.edits/27.seqid 2023-02-12 03:59:03,738 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/recovered.edits/27.seqid to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/recovered.edits/27.seqid 2023-02-12 03:59:03,738 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:59:03,738 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/recovered.edits/27.seqid to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/recovered.edits/27.seqid 2023-02-12 03:59:03,738 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:59:03,739 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/recovered.edits/27.seqid to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/recovered.edits/27.seqid 2023-02-12 03:59:03,739 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:59:03,739 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:59:03,740 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/recovered.edits/27.seqid to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/recovered.edits/27.seqid 2023-02-12 03:59:03,740 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:59:03,740 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/recovered.edits/27.seqid to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/recovered.edits/27.seqid 2023-02-12 03:59:03,741 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:59:03,741 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676174318078 regions 2023-02-12 03:59:03,741 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349 2023-02-12 03:59:03,742 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf] 2023-02-12 03:59:03,744 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f 2023-02-12 03:59:03,745 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b 2023-02-12 03:59:03,746 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 2023-02-12 03:59:03,747 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 2023-02-12 03:59:03,748 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 2023-02-12 03:59:03,749 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c 2023-02-12 03:59:03,750 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b 2023-02-12 03:59:03,751 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 2023-02-12 03:59:03,752 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 2023-02-12 03:59:03,754 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 2023-02-12 03:59:03,755 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 2023-02-12 03:59:03,757 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 2023-02-12 03:59:03,758 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349 2023-02-12 03:59:03,760 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=174, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:03,783 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676174318078 from hbase:meta 2023-02-12 03:59:03,785 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676174318078' descriptor. 2023-02-12 03:59:03,787 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=174, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:03,787 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676174318078' from region states. 2023-02-12 03:59:03,787 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174343787"}]},"ts":"9223372036854775807"} 2023-02-12 03:59:03,787 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174343787"}]},"ts":"9223372036854775807"} 2023-02-12 03:59:03,787 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174343787"}]},"ts":"9223372036854775807"} 2023-02-12 03:59:03,787 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174343787"}]},"ts":"9223372036854775807"} 2023-02-12 03:59:03,787 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174343787"}]},"ts":"9223372036854775807"} 2023-02-12 03:59:03,788 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174343787"}]},"ts":"9223372036854775807"} 2023-02-12 03:59:03,790 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-12 03:59:03,790 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => cc0e0b01138d4eae5df1e5641b484f45, NAME => 'testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => ecda59e01589f4f279b19a5420a1e608, NAME => 'testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => fcd2ed887eb9f47c07236428879982c1, NAME => 'testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 609b1a331ab507675f39b4bdbe7a382c, NAME => 'testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => e019e3f4c00b3da154d59b4085911d76, NAME => 'testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => b6894e9d4c505cdcd16894575e6b138a, NAME => 'testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.', STARTKEY => '5', ENDKEY => ''}] 2023-02-12 03:59:03,790 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676174318078' as deleted. 2023-02-12 03:59:03,790 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676174343790"}]},"ts":"9223372036854775807"} 2023-02-12 03:59:03,792 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676174318078 state from META 2023-02-12 03:59:03,793 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=174, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:03,794 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=174, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676174318078 in 96 msec 2023-02-12 03:59:03,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-12 03:59:03,961 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676174318078, procId: 174 completed 2023-02-12 03:59:03,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=175, state=RUNNABLE:CLONE_SNAPSHOT_PRE_OPERATION; CloneSnapshotProcedure (table=testRestoreSnapshot-1676174318078 snapshot=name: "snaptb1-1676174318078" table: "testRestoreSnapshot-1676174318078" creation_time: 1676174330586 type: DISABLED version: 2 ttl: 0 ) 2023-02-12 03:59:03,993 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(750): Clone snapshot=snaptb1-1676174318078 as table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:03,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-12 03:59:04,016 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/.tabledesc/.tableinfo.0000000001 2023-02-12 03:59:04,018 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676174318078" table: "testRestoreSnapshot-1676174318078" creation_time: 1676174330586 type: DISABLED version: 2 ttl: 0 2023-02-12 03:59:04,018 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,018 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,025 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,028 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,033 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,035 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,037 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,039 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,041 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,043 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,045 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,048 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,050 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,052 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:59:04,052 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:59:04,052 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:59:04,052 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:59:04,052 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:59:04,052 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:59:04,052 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=b6894e9d4c505cdcd16894575e6b138a as b6894e9d4c505cdcd16894575e6b138a in snapshot snaptb1-1676174318078 2023-02-12 03:59:04,052 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=609b1a331ab507675f39b4bdbe7a382c as 609b1a331ab507675f39b4bdbe7a382c in snapshot snaptb1-1676174318078 2023-02-12 03:59:04,052 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=cc0e0b01138d4eae5df1e5641b484f45 as cc0e0b01138d4eae5df1e5641b484f45 in snapshot snaptb1-1676174318078 2023-02-12 03:59:04,052 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=fcd2ed887eb9f47c07236428879982c1 as fcd2ed887eb9f47c07236428879982c1 in snapshot snaptb1-1676174318078 2023-02-12 03:59:04,052 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=ecda59e01589f4f279b19a5420a1e608 as ecda59e01589f4f279b19a5420a1e608 in snapshot snaptb1-1676174318078 2023-02-12 03:59:04,052 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=e019e3f4c00b3da154d59b4085911d76 as e019e3f4c00b3da154d59b4085911d76 in snapshot snaptb1-1676174318078 2023-02-12 03:59:04,054 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(7671): creating {ENCODED => b6894e9d4c505cdcd16894575e6b138a, NAME => 'testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676174318078', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp 2023-02-12 03:59:04,055 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(7671): creating {ENCODED => e019e3f4c00b3da154d59b4085911d76, NAME => 'testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676174318078', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp 2023-02-12 03:59:04,055 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(7671): creating {ENCODED => ecda59e01589f4f279b19a5420a1e608, NAME => 'testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676174318078', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp 2023-02-12 03:59:04,056 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(7671): creating {ENCODED => fcd2ed887eb9f47c07236428879982c1, NAME => 'testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676174318078', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp 2023-02-12 03:59:04,056 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(7671): creating {ENCODED => cc0e0b01138d4eae5df1e5641b484f45, NAME => 'testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676174318078', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp 2023-02-12 03:59:04,057 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(7671): creating {ENCODED => 609b1a331ab507675f39b4bdbe7a382c, NAME => 'testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676174318078', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp 2023-02-12 03:59:04,110 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:04,111 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 2e835d2ef04e4f84a4c85680200dc844 from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,115 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink f73ec6d83ce444e98c68cfc9bb0662a3 from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,127 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:04,128 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:04,128 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 25b90241f56e47b781d04ecdbb8f5bd5 from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,128 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 6ae5fbf6b3ca4a899b9e29e73711608c from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,135 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:04,135 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1603): Closing fcd2ed887eb9f47c07236428879982c1, disabling compactions & flushes 2023-02-12 03:59:04,135 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 027aff06c1054aaaa903715be8404fab from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,136 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:59:04,136 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:59:04,136 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. after waiting 0 ms 2023-02-12 03:59:04,136 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:59:04,136 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:59:04,136 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1557): Region close journal for fcd2ed887eb9f47c07236428879982c1: Waiting for close lock at 1676174344135Disabling compacts and flushes for region at 1676174344135Disabling writes for close at 1676174344136 (+1 ms)Writing region close event to WAL at 1676174344136Closed at 1676174344136 2023-02-12 03:59:04,136 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink f75b9c751c584507ac2da36d4b2eb053 from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,143 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1603): Closing ecda59e01589f4f279b19a5420a1e608, disabling compactions & flushes 2023-02-12 03:59:04,143 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:59:04,143 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:59:04,143 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. after waiting 0 ms 2023-02-12 03:59:04,143 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:59:04,143 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:59:04,143 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1557): Region close journal for ecda59e01589f4f279b19a5420a1e608: Waiting for close lock at 1676174344143Disabling compacts and flushes for region at 1676174344143Disabling writes for close at 1676174344143Writing region close event to WAL at 1676174344143Closed at 1676174344143 2023-02-12 03:59:04,146 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:04,146 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 541575adc02c4606b85cbfea9970d250 from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,147 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink edd4e077572a45bcac8218359e3d1f91 from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,147 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:04,147 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 1029f052cfe34ddb8506a2f181ab2423 from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,149 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 10e6966d5cc341688a6163ff0d1d3493 from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,153 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 6148ef2edb69420e9208fff789b259f0 from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,154 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1603): Closing b6894e9d4c505cdcd16894575e6b138a, disabling compactions & flushes 2023-02-12 03:59:04,154 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:59:04,154 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:59:04,154 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. after waiting 0 ms 2023-02-12 03:59:04,154 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:59:04,155 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:59:04,153 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink de3f19ac4a9247fbac10bbe6eed0d5d1 from cloned region in snapshot snaptb1-1676174318078 to table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:04,155 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1557): Region close journal for b6894e9d4c505cdcd16894575e6b138a: Waiting for close lock at 1676174344154Disabling compacts and flushes for region at 1676174344154Disabling writes for close at 1676174344154Writing region close event to WAL at 1676174344155 (+1 ms)Closed at 1676174344155 2023-02-12 03:59:04,155 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1603): Closing cc0e0b01138d4eae5df1e5641b484f45, disabling compactions & flushes 2023-02-12 03:59:04,155 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:59:04,155 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:59:04,155 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. after waiting 0 ms 2023-02-12 03:59:04,155 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:59:04,155 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:59:04,155 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1557): Region close journal for cc0e0b01138d4eae5df1e5641b484f45: Waiting for close lock at 1676174344155Disabling compacts and flushes for region at 1676174344155Disabling writes for close at 1676174344155Writing region close event to WAL at 1676174344155Closed at 1676174344155 2023-02-12 03:59:04,161 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1603): Closing e019e3f4c00b3da154d59b4085911d76, disabling compactions & flushes 2023-02-12 03:59:04,162 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:59:04,162 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:59:04,162 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. after waiting 0 ms 2023-02-12 03:59:04,162 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:59:04,162 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:59:04,162 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1557): Region close journal for e019e3f4c00b3da154d59b4085911d76: Waiting for close lock at 1676174344161Disabling compacts and flushes for region at 1676174344161Disabling writes for close at 1676174344162 (+1 ms)Writing region close event to WAL at 1676174344162Closed at 1676174344162 2023-02-12 03:59:04,162 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1603): Closing 609b1a331ab507675f39b4bdbe7a382c, disabling compactions & flushes 2023-02-12 03:59:04,162 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:59:04,162 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:59:04,162 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. after waiting 0 ms 2023-02-12 03:59:04,162 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:59:04,162 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:59:04,162 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1557): Region close journal for 609b1a331ab507675f39b4bdbe7a382c: Waiting for close lock at 1676174344162Disabling compacts and flushes for region at 1676174344162Disabling writes for close at 1676174344162Writing region close event to WAL at 1676174344162Closed at 1676174344162 2023-02-12 03:59:04,162 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676174318078" table: "testRestoreSnapshot-1676174318078" creation_time: 1676174330586 type: DISABLED version: 2 ttl: 0 2023-02-12 03:59:04,162 INFO [PEWorker-3] procedure.CloneSnapshotProcedure$1(411): Clone snapshot=snaptb1-1676174318078 on table=testRestoreSnapshot-1676174318078 completed! 2023-02-12 03:59:04,167 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174344167"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174344167"}]},"ts":"1676174344167"} 2023-02-12 03:59:04,168 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174344167"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174344167"}]},"ts":"1676174344167"} 2023-02-12 03:59:04,168 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174344167"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174344167"}]},"ts":"1676174344167"} 2023-02-12 03:59:04,168 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174344167"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174344167"}]},"ts":"1676174344167"} 2023-02-12 03:59:04,168 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174344167"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174344167"}]},"ts":"1676174344167"} 2023-02-12 03:59:04,168 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174344167"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174344167"}]},"ts":"1676174344167"} 2023-02-12 03:59:04,170 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-12 03:59:04,171 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174344171"}]},"ts":"1676174344171"} 2023-02-12 03:59:04,172 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=ENABLING in hbase:meta 2023-02-12 03:59:04,175 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-02-12 03:59:04,176 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-12 03:59:04,176 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-12 03:59:04,176 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-12 03:59:04,176 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-12 03:59:04,176 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN}, {pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN}, {pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN}, {pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN}, {pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN}, {pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN}] 2023-02-12 03:59:04,178 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN 2023-02-12 03:59:04,178 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN 2023-02-12 03:59:04,178 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN 2023-02-12 03:59:04,178 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN 2023-02-12 03:59:04,178 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,35783,1676174315451; forceNewPlan=false, retain=false 2023-02-12 03:59:04,179 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,35785,1676174315481; forceNewPlan=false, retain=false 2023-02-12 03:59:04,179 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,42205,1676174315413; forceNewPlan=false, retain=false 2023-02-12 03:59:04,179 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,35783,1676174315451; forceNewPlan=false, retain=false 2023-02-12 03:59:04,179 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN 2023-02-12 03:59:04,179 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN 2023-02-12 03:59:04,179 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,42205,1676174315413; forceNewPlan=false, retain=false 2023-02-12 03:59:04,180 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,35785,1676174315481; forceNewPlan=false, retain=false 2023-02-12 03:59:04,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-12 03:59:04,329 INFO [jenkins-hbase20:37305] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-12 03:59:04,334 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:04,334 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:04,334 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:04,334 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:04,334 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174344333"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174344333"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174344333"}]},"ts":"1676174344333"} 2023-02-12 03:59:04,334 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174344333"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174344333"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174344333"}]},"ts":"1676174344333"} 2023-02-12 03:59:04,334 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:04,334 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174344333"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174344333"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174344333"}]},"ts":"1676174344333"} 2023-02-12 03:59:04,334 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174344333"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174344333"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174344333"}]},"ts":"1676174344333"} 2023-02-12 03:59:04,334 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174344333"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174344333"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174344333"}]},"ts":"1676174344333"} 2023-02-12 03:59:04,336 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=182, ppid=178, state=RUNNABLE; OpenRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:59:04,339 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=183, ppid=181, state=RUNNABLE; OpenRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:59:04,341 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=184, ppid=180, state=RUNNABLE; OpenRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:59:04,341 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=185, ppid=177, state=RUNNABLE; OpenRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:59:04,344 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:04,344 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174344344"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174344344"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174344344"}]},"ts":"1676174344344"} 2023-02-12 03:59:04,344 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=186, ppid=176, state=RUNNABLE; OpenRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:59:04,345 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=187, ppid=179, state=RUNNABLE; OpenRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:59:04,493 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:59:04,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => fcd2ed887eb9f47c07236428879982c1, NAME => 'testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.', STARTKEY => '2', ENDKEY => '3'} 2023-02-12 03:59:04,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:59:04,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:04,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:59:04,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:59:04,495 INFO [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:59:04,497 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:59:04,497 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:59:04,497 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => cc0e0b01138d4eae5df1e5641b484f45, NAME => 'testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.', STARTKEY => '', ENDKEY => '1'} 2023-02-12 03:59:04,497 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ecda59e01589f4f279b19a5420a1e608, NAME => 'testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.', STARTKEY => '1', ENDKEY => '2'} 2023-02-12 03:59:04,497 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:59:04,497 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:59:04,497 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:04,497 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:04,497 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:59:04,497 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:59:04,497 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:59:04,497 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:59:04,499 INFO [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:59:04,499 INFO [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:59:04,499 DEBUG [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf 2023-02-12 03:59:04,499 DEBUG [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf 2023-02-12 03:59:04,499 DEBUG [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf 2023-02-12 03:59:04,499 INFO [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fcd2ed887eb9f47c07236428879982c1 columnFamilyName cf 2023-02-12 03:59:04,499 DEBUG [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf 2023-02-12 03:59:04,500 INFO [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ecda59e01589f4f279b19a5420a1e608 columnFamilyName cf 2023-02-12 03:59:04,500 DEBUG [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf 2023-02-12 03:59:04,500 DEBUG [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf 2023-02-12 03:59:04,500 INFO [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cc0e0b01138d4eae5df1e5641b484f45 columnFamilyName cf 2023-02-12 03:59:04,519 DEBUG [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/testRestoreSnapshot-1676174318078=fcd2ed887eb9f47c07236428879982c1-2e835d2ef04e4f84a4c85680200dc844 2023-02-12 03:59:04,519 DEBUG [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/testRestoreSnapshot-1676174318078=cc0e0b01138d4eae5df1e5641b484f45-027aff06c1054aaaa903715be8404fab 2023-02-12 03:59:04,520 DEBUG [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/testRestoreSnapshot-1676174318078=ecda59e01589f4f279b19a5420a1e608-25b90241f56e47b781d04ecdbb8f5bd5 2023-02-12 03:59:04,532 DEBUG [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/testRestoreSnapshot-1676174318078=fcd2ed887eb9f47c07236428879982c1-f73ec6d83ce444e98c68cfc9bb0662a3 2023-02-12 03:59:04,535 INFO [StoreOpener-fcd2ed887eb9f47c07236428879982c1-1] regionserver.HStore(310): Store=fcd2ed887eb9f47c07236428879982c1/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:04,535 DEBUG [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/testRestoreSnapshot-1676174318078=cc0e0b01138d4eae5df1e5641b484f45-10e6966d5cc341688a6163ff0d1d3493 2023-02-12 03:59:04,535 DEBUG [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/testRestoreSnapshot-1676174318078=ecda59e01589f4f279b19a5420a1e608-f75b9c751c584507ac2da36d4b2eb053 2023-02-12 03:59:04,536 INFO [StoreOpener-cc0e0b01138d4eae5df1e5641b484f45-1] regionserver.HStore(310): Store=cc0e0b01138d4eae5df1e5641b484f45/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:04,536 INFO [StoreOpener-ecda59e01589f4f279b19a5420a1e608-1] regionserver.HStore(310): Store=ecda59e01589f4f279b19a5420a1e608/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:04,536 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:59:04,536 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:59:04,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:59:04,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:59:04,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:59:04,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:59:04,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:59:04,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:59:04,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:59:04,542 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-12 03:59:04,542 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened cc0e0b01138d4eae5df1e5641b484f45; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73468915, jitterRate=0.09477214515209198}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:04,542 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for cc0e0b01138d4eae5df1e5641b484f45: Running coprocessor pre-open hook at 1676174344498Writing region info on filesystem at 1676174344498Initializing all the Stores at 1676174344498Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174344498Cleaning up temporary data from old regions at 1676174344537 (+39 ms)Cleaning up detritus from prior splits at 1676174344538 (+1 ms)Running coprocessor post-open hooks at 1676174344542 (+4 ms)Region opened successfully at 1676174344542 2023-02-12 03:59:04,543 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45., pid=186, masterSystemTime=1676174344495 2023-02-12 03:59:04,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-12 03:59:04,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-12 03:59:04,544 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened ecda59e01589f4f279b19a5420a1e608; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72709433, jitterRate=0.08345498144626617}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:04,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for ecda59e01589f4f279b19a5420a1e608: Running coprocessor pre-open hook at 1676174344498Writing region info on filesystem at 1676174344498Initializing all the Stores at 1676174344498Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174344498Cleaning up temporary data from old regions at 1676174344537 (+39 ms)Cleaning up detritus from prior splits at 1676174344538 (+1 ms)Running coprocessor post-open hooks at 1676174344544 (+6 ms)Region opened successfully at 1676174344544 2023-02-12 03:59:04,545 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened fcd2ed887eb9f47c07236428879982c1; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67774485, jitterRate=0.00991852581501007}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:04,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for fcd2ed887eb9f47c07236428879982c1: Running coprocessor pre-open hook at 1676174344494Writing region info on filesystem at 1676174344494Initializing all the Stores at 1676174344494Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174344494Cleaning up temporary data from old regions at 1676174344537 (+43 ms)Cleaning up detritus from prior splits at 1676174344538 (+1 ms)Running coprocessor post-open hooks at 1676174344545 (+7 ms)Region opened successfully at 1676174344545 2023-02-12 03:59:04,545 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608., pid=185, masterSystemTime=1676174344495 2023-02-12 03:59:04,545 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1., pid=182, masterSystemTime=1676174344490 2023-02-12 03:59:04,545 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:59:04,545 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:59:04,546 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:59:04,546 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e019e3f4c00b3da154d59b4085911d76, NAME => 'testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.', STARTKEY => '4', ENDKEY => '5'} 2023-02-12 03:59:04,546 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:59:04,546 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:04,546 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:59:04,546 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:59:04,547 INFO [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:59:04,548 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:04,548 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174344548"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174344548"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174344548"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174344548"}]},"ts":"1676174344548"} 2023-02-12 03:59:04,549 DEBUG [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf 2023-02-12 03:59:04,549 DEBUG [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf 2023-02-12 03:59:04,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:59:04,549 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:04,549 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:59:04,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:59:04,549 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:59:04,549 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174344549"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174344549"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174344549"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174344549"}]},"ts":"1676174344549"} 2023-02-12 03:59:04,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => b6894e9d4c505cdcd16894575e6b138a, NAME => 'testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.', STARTKEY => '5', ENDKEY => ''} 2023-02-12 03:59:04,550 INFO [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e019e3f4c00b3da154d59b4085911d76 columnFamilyName cf 2023-02-12 03:59:04,549 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:59:04,550 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:59:04,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:59:04,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 609b1a331ab507675f39b4bdbe7a382c, NAME => 'testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.', STARTKEY => '3', ENDKEY => '4'} 2023-02-12 03:59:04,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:04,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:59:04,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:59:04,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676174318078 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:59:04,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:04,550 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:04,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:59:04,550 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174344550"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174344550"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174344550"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174344550"}]},"ts":"1676174344550"} 2023-02-12 03:59:04,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:59:04,552 INFO [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:59:04,552 INFO [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:59:04,553 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=186, resume processing ppid=176 2023-02-12 03:59:04,553 DEBUG [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf 2023-02-12 03:59:04,553 DEBUG [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf 2023-02-12 03:59:04,553 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=186, ppid=176, state=SUCCESS; OpenRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35785,1676174315481 in 205 msec 2023-02-12 03:59:04,553 DEBUG [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf 2023-02-12 03:59:04,553 DEBUG [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf 2023-02-12 03:59:04,553 INFO [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b6894e9d4c505cdcd16894575e6b138a columnFamilyName cf 2023-02-12 03:59:04,555 INFO [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 609b1a331ab507675f39b4bdbe7a382c columnFamilyName cf 2023-02-12 03:59:04,556 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=182, resume processing ppid=178 2023-02-12 03:59:04,556 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=176, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, ASSIGN in 377 msec 2023-02-12 03:59:04,556 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=182, ppid=178, state=SUCCESS; OpenRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,42205,1676174315413 in 216 msec 2023-02-12 03:59:04,556 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=185, resume processing ppid=177 2023-02-12 03:59:04,556 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=185, ppid=177, state=SUCCESS; OpenRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,35783,1676174315451 in 212 msec 2023-02-12 03:59:04,559 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=178, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, ASSIGN in 380 msec 2023-02-12 03:59:04,560 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=177, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, ASSIGN in 380 msec 2023-02-12 03:59:04,576 DEBUG [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/testRestoreSnapshot-1676174318078=e019e3f4c00b3da154d59b4085911d76-541575adc02c4606b85cbfea9970d250 2023-02-12 03:59:04,582 DEBUG [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/testRestoreSnapshot-1676174318078=b6894e9d4c505cdcd16894575e6b138a-6ae5fbf6b3ca4a899b9e29e73711608c 2023-02-12 03:59:04,587 DEBUG [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/testRestoreSnapshot-1676174318078=b6894e9d4c505cdcd16894575e6b138a-edd4e077572a45bcac8218359e3d1f91 2023-02-12 03:59:04,587 INFO [StoreOpener-b6894e9d4c505cdcd16894575e6b138a-1] regionserver.HStore(310): Store=b6894e9d4c505cdcd16894575e6b138a/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:04,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:59:04,588 DEBUG [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/testRestoreSnapshot-1676174318078=609b1a331ab507675f39b4bdbe7a382c-1029f052cfe34ddb8506a2f181ab2423 2023-02-12 03:59:04,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:59:04,588 DEBUG [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/testRestoreSnapshot-1676174318078=e019e3f4c00b3da154d59b4085911d76-6148ef2edb69420e9208fff789b259f0 2023-02-12 03:59:04,589 INFO [StoreOpener-e019e3f4c00b3da154d59b4085911d76-1] regionserver.HStore(310): Store=e019e3f4c00b3da154d59b4085911d76/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:04,589 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:59:04,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:59:04,591 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:59:04,592 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:59:04,593 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-12 03:59:04,593 DEBUG [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/testRestoreSnapshot-1676174318078=609b1a331ab507675f39b4bdbe7a382c-de3f19ac4a9247fbac10bbe6eed0d5d1 2023-02-12 03:59:04,593 INFO [StoreOpener-609b1a331ab507675f39b4bdbe7a382c-1] regionserver.HStore(310): Store=609b1a331ab507675f39b4bdbe7a382c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:04,593 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened b6894e9d4c505cdcd16894575e6b138a; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74894917, jitterRate=0.11602123081684113}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:04,593 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for b6894e9d4c505cdcd16894575e6b138a: Running coprocessor pre-open hook at 1676174344550Writing region info on filesystem at 1676174344550Initializing all the Stores at 1676174344551 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174344551Cleaning up temporary data from old regions at 1676174344589 (+38 ms)Cleaning up detritus from prior splits at 1676174344589Running coprocessor post-open hooks at 1676174344593 (+4 ms)Region opened successfully at 1676174344593 2023-02-12 03:59:04,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-12 03:59:04,594 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a., pid=183, masterSystemTime=1676174344490 2023-02-12 03:59:04,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:59:04,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:59:04,594 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened e019e3f4c00b3da154d59b4085911d76; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73771742, jitterRate=0.09928461909294128}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:04,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for e019e3f4c00b3da154d59b4085911d76: Running coprocessor pre-open hook at 1676174344546Writing region info on filesystem at 1676174344546Initializing all the Stores at 1676174344547 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174344547Cleaning up temporary data from old regions at 1676174344590 (+43 ms)Cleaning up detritus from prior splits at 1676174344591 (+1 ms)Running coprocessor post-open hooks at 1676174344594 (+3 ms)Region opened successfully at 1676174344594 2023-02-12 03:59:04,595 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76., pid=184, masterSystemTime=1676174344495 2023-02-12 03:59:04,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:59:04,596 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:59:04,597 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:04,597 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174344596"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174344596"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174344596"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174344596"}]},"ts":"1676174344596"} 2023-02-12 03:59:04,597 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:59:04,597 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:59:04,598 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:04,598 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174344598"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174344598"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174344598"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174344598"}]},"ts":"1676174344598"} 2023-02-12 03:59:04,598 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:59:04,601 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=183, resume processing ppid=181 2023-02-12 03:59:04,601 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=183, ppid=181, state=SUCCESS; OpenRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,42205,1676174315413 in 261 msec 2023-02-12 03:59:04,602 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=184, resume processing ppid=180 2023-02-12 03:59:04,602 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=181, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, ASSIGN in 425 msec 2023-02-12 03:59:04,602 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=184, ppid=180, state=SUCCESS; OpenRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,35785,1676174315481 in 258 msec 2023-02-12 03:59:04,603 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=180, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, ASSIGN in 426 msec 2023-02-12 03:59:04,607 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-12 03:59:04,608 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 609b1a331ab507675f39b4bdbe7a382c; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64609433, jitterRate=-0.03724442422389984}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:04,608 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 609b1a331ab507675f39b4bdbe7a382c: Running coprocessor pre-open hook at 1676174344551Writing region info on filesystem at 1676174344551Initializing all the Stores at 1676174344551Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174344551Cleaning up temporary data from old regions at 1676174344595 (+44 ms)Cleaning up detritus from prior splits at 1676174344595Running coprocessor post-open hooks at 1676174344608 (+13 ms)Region opened successfully at 1676174344608 2023-02-12 03:59:04,609 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c., pid=187, masterSystemTime=1676174344495 2023-02-12 03:59:04,610 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:59:04,610 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:59:04,610 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:04,610 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174344610"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174344610"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174344610"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174344610"}]},"ts":"1676174344610"} 2023-02-12 03:59:04,613 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=187, resume processing ppid=179 2023-02-12 03:59:04,613 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=187, ppid=179, state=SUCCESS; OpenRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451 in 267 msec 2023-02-12 03:59:04,614 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=179, resume processing ppid=175 2023-02-12 03:59:04,614 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=179, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, ASSIGN in 437 msec 2023-02-12 03:59:04,615 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174344615"}]},"ts":"1676174344615"} 2023-02-12 03:59:04,617 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=ENABLED in hbase:meta 2023-02-12 03:59:04,621 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=175, state=SUCCESS; CloneSnapshotProcedure (table=testRestoreSnapshot-1676174318078 snapshot=name: "snaptb1-1676174318078" table: "testRestoreSnapshot-1676174318078" creation_time: 1676174330586 type: DISABLED version: 2 ttl: 0 ) in 633 msec 2023-02-12 03:59:04,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-12 03:59:04,750 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676174318078, procId: 175 completed 2023-02-12 03:59:04,751 DEBUG [Listener at localhost.localdomain/46483] zookeeper.ReadOnlyZKClient(139): Connect 0x79efc34c to 127.0.0.1:50368 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 03:59:04,765 DEBUG [Listener at localhost.localdomain/46483] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4dd35f79, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 03:59:04,769 DEBUG [hconnection-0x383008f9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:59:04,773 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60368, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:59:04,782 DEBUG [hconnection-0x383008f9-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:59:04,788 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47348, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:59:04,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:04,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,157 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-12 03:59:05,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,584 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-12 03:59:05,616 DEBUG [hconnection-0x383008f9-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:59:05,634 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:57456, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:59:05,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:05,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,231 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-12 03:59:06,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,691 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-12 03:59:06,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:06,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,027 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-12 03:59:07,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:07,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:08,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,122 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-12 03:59:09,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,190 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27ddad26 to 127.0.0.1:50368 2023-02-12 03:59:09,191 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:59:09,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:09,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:10,277 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676174318078 2023-02-12 03:59:10,277 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$11(2405): Client=jenkins//148.251.75.209 disable testRestoreSnapshot-1676174318078 2023-02-12 03:59:10,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=188, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:10,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-12 03:59:10,285 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174350285"}]},"ts":"1676174350285"} 2023-02-12 03:59:10,288 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=DISABLING in hbase:meta 2023-02-12 03:59:10,291 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676174318078 to state=DISABLING 2023-02-12 03:59:10,292 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, UNASSIGN}, {pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, UNASSIGN}, {pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, UNASSIGN}, {pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, UNASSIGN}, {pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, UNASSIGN}, {pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, UNASSIGN}] 2023-02-12 03:59:10,294 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, UNASSIGN 2023-02-12 03:59:10,295 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, UNASSIGN 2023-02-12 03:59:10,295 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, UNASSIGN 2023-02-12 03:59:10,296 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:10,296 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:10,296 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174350296"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174350296"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174350296"}]},"ts":"1676174350296"} 2023-02-12 03:59:10,296 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:10,296 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174350296"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174350296"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174350296"}]},"ts":"1676174350296"} 2023-02-12 03:59:10,296 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174350296"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174350296"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174350296"}]},"ts":"1676174350296"} 2023-02-12 03:59:10,297 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, UNASSIGN 2023-02-12 03:59:10,297 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, UNASSIGN 2023-02-12 03:59:10,298 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=195, ppid=192, state=RUNNABLE; CloseRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:59:10,298 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=196, ppid=189, state=RUNNABLE; CloseRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:59:10,299 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:10,299 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174350299"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174350299"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174350299"}]},"ts":"1676174350299"} 2023-02-12 03:59:10,299 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=197, ppid=191, state=RUNNABLE; CloseRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:59:10,300 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:10,300 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174350300"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174350300"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174350300"}]},"ts":"1676174350300"} 2023-02-12 03:59:10,305 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=198, ppid=194, state=RUNNABLE; CloseRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:59:10,309 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=199, ppid=190, state=RUNNABLE; CloseRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:59:10,312 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, UNASSIGN 2023-02-12 03:59:10,314 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:10,314 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174350314"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174350314"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174350314"}]},"ts":"1676174350314"} 2023-02-12 03:59:10,316 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=200, ppid=193, state=RUNNABLE; CloseRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:59:10,457 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:59:10,457 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:59:10,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing b6894e9d4c505cdcd16894575e6b138a, disabling compactions & flushes 2023-02-12 03:59:10,458 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:59:10,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing cc0e0b01138d4eae5df1e5641b484f45, disabling compactions & flushes 2023-02-12 03:59:10,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:59:10,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. after waiting 0 ms 2023-02-12 03:59:10,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:59:10,459 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:59:10,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:59:10,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. after waiting 0 ms 2023-02-12 03:59:10,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:59:10,462 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:59:10,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing ecda59e01589f4f279b19a5420a1e608, disabling compactions & flushes 2023-02-12 03:59:10,466 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:59:10,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:59:10,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. after waiting 0 ms 2023-02-12 03:59:10,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:59:10,512 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-12 03:59:10,516 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608. 2023-02-12 03:59:10,516 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for ecda59e01589f4f279b19a5420a1e608: Waiting for close lock at 1676174350462Running coprocessor pre-close hooks at 1676174350462Disabling compacts and flushes for region at 1676174350462Disabling writes for close at 1676174350466 (+4 ms)Writing region close event to WAL at 1676174350504 (+38 ms)Running coprocessor post-close hooks at 1676174350513 (+9 ms)Closed at 1676174350516 (+3 ms) 2023-02-12 03:59:10,518 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:59:10,518 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:59:10,519 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=ecda59e01589f4f279b19a5420a1e608, regionState=CLOSED 2023-02-12 03:59:10,519 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174350519"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174350519"}]},"ts":"1676174350519"} 2023-02-12 03:59:10,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 609b1a331ab507675f39b4bdbe7a382c, disabling compactions & flushes 2023-02-12 03:59:10,521 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:59:10,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:59:10,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. after waiting 0 ms 2023-02-12 03:59:10,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:59:10,525 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=199, resume processing ppid=190 2023-02-12 03:59:10,525 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=199, ppid=190, state=SUCCESS; CloseRegionProcedure ecda59e01589f4f279b19a5420a1e608, server=jenkins-hbase20.apache.org,35783,1676174315451 in 213 msec 2023-02-12 03:59:10,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-12 03:59:10,528 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=190, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=ecda59e01589f4f279b19a5420a1e608, UNASSIGN in 233 msec 2023-02-12 03:59:10,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-12 03:59:10,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-12 03:59:10,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45. 2023-02-12 03:59:10,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for cc0e0b01138d4eae5df1e5641b484f45: Waiting for close lock at 1676174350457Running coprocessor pre-close hooks at 1676174350457Disabling compacts and flushes for region at 1676174350457Disabling writes for close at 1676174350459 (+2 ms)Writing region close event to WAL at 1676174350507 (+48 ms)Running coprocessor post-close hooks at 1676174350535 (+28 ms)Closed at 1676174350537 (+2 ms) 2023-02-12 03:59:10,539 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:59:10,539 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:59:10,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing e019e3f4c00b3da154d59b4085911d76, disabling compactions & flushes 2023-02-12 03:59:10,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:59:10,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:59:10,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. after waiting 0 ms 2023-02-12 03:59:10,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:59:10,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a. 2023-02-12 03:59:10,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for b6894e9d4c505cdcd16894575e6b138a: Waiting for close lock at 1676174350457Running coprocessor pre-close hooks at 1676174350457Disabling compacts and flushes for region at 1676174350457Disabling writes for close at 1676174350459 (+2 ms)Writing region close event to WAL at 1676174350509 (+50 ms)Running coprocessor post-close hooks at 1676174350537 (+28 ms)Closed at 1676174350541 (+4 ms) 2023-02-12 03:59:10,542 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=cc0e0b01138d4eae5df1e5641b484f45, regionState=CLOSED 2023-02-12 03:59:10,542 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174350542"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174350542"}]},"ts":"1676174350542"} 2023-02-12 03:59:10,543 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:59:10,543 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:59:10,544 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=b6894e9d4c505cdcd16894575e6b138a, regionState=CLOSED 2023-02-12 03:59:10,544 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676174350544"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174350544"}]},"ts":"1676174350544"} 2023-02-12 03:59:10,548 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing fcd2ed887eb9f47c07236428879982c1, disabling compactions & flushes 2023-02-12 03:59:10,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:59:10,548 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:59:10,548 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. after waiting 0 ms 2023-02-12 03:59:10,549 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:59:10,553 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=196, resume processing ppid=189 2023-02-12 03:59:10,553 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=196, ppid=189, state=SUCCESS; CloseRegionProcedure cc0e0b01138d4eae5df1e5641b484f45, server=jenkins-hbase20.apache.org,35785,1676174315481 in 246 msec 2023-02-12 03:59:10,554 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-12 03:59:10,558 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=198, resume processing ppid=194 2023-02-12 03:59:10,558 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=189, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=cc0e0b01138d4eae5df1e5641b484f45, UNASSIGN in 261 msec 2023-02-12 03:59:10,558 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=198, ppid=194, state=SUCCESS; CloseRegionProcedure b6894e9d4c505cdcd16894575e6b138a, server=jenkins-hbase20.apache.org,42205,1676174315413 in 245 msec 2023-02-12 03:59:10,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-12 03:59:10,560 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c. 2023-02-12 03:59:10,561 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 609b1a331ab507675f39b4bdbe7a382c: Waiting for close lock at 1676174350518Running coprocessor pre-close hooks at 1676174350518Disabling compacts and flushes for region at 1676174350518Disabling writes for close at 1676174350521 (+3 ms)Writing region close event to WAL at 1676174350535 (+14 ms)Running coprocessor post-close hooks at 1676174350554 (+19 ms)Closed at 1676174350560 (+6 ms) 2023-02-12 03:59:10,569 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=194, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=b6894e9d4c505cdcd16894575e6b138a, UNASSIGN in 266 msec 2023-02-12 03:59:10,569 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76. 2023-02-12 03:59:10,570 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:59:10,570 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for e019e3f4c00b3da154d59b4085911d76: Waiting for close lock at 1676174350540Running coprocessor pre-close hooks at 1676174350540Disabling compacts and flushes for region at 1676174350540Disabling writes for close at 1676174350541 (+1 ms)Writing region close event to WAL at 1676174350551 (+10 ms)Running coprocessor post-close hooks at 1676174350567 (+16 ms)Closed at 1676174350569 (+2 ms) 2023-02-12 03:59:10,570 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=609b1a331ab507675f39b4bdbe7a382c, regionState=CLOSED 2023-02-12 03:59:10,570 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174350570"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174350570"}]},"ts":"1676174350570"} 2023-02-12 03:59:10,572 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:59:10,573 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=e019e3f4c00b3da154d59b4085911d76, regionState=CLOSED 2023-02-12 03:59:10,573 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174350573"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174350573"}]},"ts":"1676174350573"} 2023-02-12 03:59:10,575 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=195, resume processing ppid=192 2023-02-12 03:59:10,575 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=195, ppid=192, state=SUCCESS; CloseRegionProcedure 609b1a331ab507675f39b4bdbe7a382c, server=jenkins-hbase20.apache.org,35783,1676174315451 in 274 msec 2023-02-12 03:59:10,577 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=192, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=609b1a331ab507675f39b4bdbe7a382c, UNASSIGN in 283 msec 2023-02-12 03:59:10,578 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=200, resume processing ppid=193 2023-02-12 03:59:10,578 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=200, ppid=193, state=SUCCESS; CloseRegionProcedure e019e3f4c00b3da154d59b4085911d76, server=jenkins-hbase20.apache.org,35785,1676174315481 in 260 msec 2023-02-12 03:59:10,579 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=193, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=e019e3f4c00b3da154d59b4085911d76, UNASSIGN in 286 msec 2023-02-12 03:59:10,580 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-12 03:59:10,581 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1. 2023-02-12 03:59:10,581 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for fcd2ed887eb9f47c07236428879982c1: Waiting for close lock at 1676174350543Running coprocessor pre-close hooks at 1676174350543Disabling compacts and flushes for region at 1676174350543Disabling writes for close at 1676174350549 (+6 ms)Writing region close event to WAL at 1676174350574 (+25 ms)Running coprocessor post-close hooks at 1676174350581 (+7 ms)Closed at 1676174350581 2023-02-12 03:59:10,583 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:59:10,584 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=fcd2ed887eb9f47c07236428879982c1, regionState=CLOSED 2023-02-12 03:59:10,584 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676174350584"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174350584"}]},"ts":"1676174350584"} 2023-02-12 03:59:10,587 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=197, resume processing ppid=191 2023-02-12 03:59:10,587 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=197, ppid=191, state=SUCCESS; CloseRegionProcedure fcd2ed887eb9f47c07236428879982c1, server=jenkins-hbase20.apache.org,42205,1676174315413 in 286 msec 2023-02-12 03:59:10,590 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=191, resume processing ppid=188 2023-02-12 03:59:10,590 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=191, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676174318078, region=fcd2ed887eb9f47c07236428879982c1, UNASSIGN in 295 msec 2023-02-12 03:59:10,591 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174350591"}]},"ts":"1676174350591"} 2023-02-12 03:59:10,592 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676174318078, state=DISABLED in hbase:meta 2023-02-12 03:59:10,594 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676174318078 to state=DISABLED 2023-02-12 03:59:10,596 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=188, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676174318078 in 318 msec 2023-02-12 03:59:11,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-12 03:59:11,039 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676174318078, procId: 188 completed 2023-02-12 03:59:11,041 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$5(2215): Client=jenkins//148.251.75.209 delete testRestoreSnapshot-1676174318078 2023-02-12 03:59:11,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:11,044 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:11,045 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=201, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:11,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-12 03:59:11,058 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:59:11,058 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:59:11,058 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:59:11,058 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:59:11,058 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:59:11,058 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:59:11,060 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf, FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/recovered.edits] 2023-02-12 03:59:11,060 DEBUG [HFileArchiver-10] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf, FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/recovered.edits] 2023-02-12 03:59:11,060 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf, FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/recovered.edits] 2023-02-12 03:59:11,060 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf, FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/recovered.edits] 2023-02-12 03:59:11,060 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf, FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/recovered.edits] 2023-02-12 03:59:11,061 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf, FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/recovered.edits] 2023-02-12 03:59:11,064 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/testRestoreSnapshot-1676174318078=609b1a331ab507675f39b4bdbe7a382c-1029f052cfe34ddb8506a2f181ab2423 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/testRestoreSnapshot-1676174318078=609b1a331ab507675f39b4bdbe7a382c-1029f052cfe34ddb8506a2f181ab2423. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343723 currentMtime: 1676174344153 2023-02-12 03:59:11,065 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/testRestoreSnapshot-1676174318078=b6894e9d4c505cdcd16894575e6b138a-6ae5fbf6b3ca4a899b9e29e73711608c already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/testRestoreSnapshot-1676174318078=b6894e9d4c505cdcd16894575e6b138a-6ae5fbf6b3ca4a899b9e29e73711608c. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343723 currentMtime: 1676174344147 2023-02-12 03:59:11,065 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/testRestoreSnapshot-1676174318078=609b1a331ab507675f39b4bdbe7a382c-1029f052cfe34ddb8506a2f181ab2423 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/testRestoreSnapshot-1676174318078=609b1a331ab507675f39b4bdbe7a382c-1029f052cfe34ddb8506a2f181ab2423.1676174351060. 2023-02-12 03:59:11,066 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/testRestoreSnapshot-1676174318078=e019e3f4c00b3da154d59b4085911d76-541575adc02c4606b85cbfea9970d250 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/testRestoreSnapshot-1676174318078=e019e3f4c00b3da154d59b4085911d76-541575adc02c4606b85cbfea9970d250. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343723 currentMtime: 1676174344153 2023-02-12 03:59:11,066 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/testRestoreSnapshot-1676174318078=e019e3f4c00b3da154d59b4085911d76-541575adc02c4606b85cbfea9970d250 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/testRestoreSnapshot-1676174318078=e019e3f4c00b3da154d59b4085911d76-541575adc02c4606b85cbfea9970d250.1676174351061. 2023-02-12 03:59:11,066 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/testRestoreSnapshot-1676174318078=b6894e9d4c505cdcd16894575e6b138a-6ae5fbf6b3ca4a899b9e29e73711608c to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/testRestoreSnapshot-1676174318078=b6894e9d4c505cdcd16894575e6b138a-6ae5fbf6b3ca4a899b9e29e73711608c.1676174351060. 2023-02-12 03:59:11,067 ERROR [HFileArchiver-10] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/testRestoreSnapshot-1676174318078=ecda59e01589f4f279b19a5420a1e608-25b90241f56e47b781d04ecdbb8f5bd5 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/testRestoreSnapshot-1676174318078=ecda59e01589f4f279b19a5420a1e608-25b90241f56e47b781d04ecdbb8f5bd5. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343724 currentMtime: 1676174344135 2023-02-12 03:59:11,067 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/testRestoreSnapshot-1676174318078=e019e3f4c00b3da154d59b4085911d76-541575adc02c4606b85cbfea9970d250 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/testRestoreSnapshot-1676174318078=e019e3f4c00b3da154d59b4085911d76-541575adc02c4606b85cbfea9970d250 2023-02-12 03:59:11,067 ERROR [HFileArchiver-9] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/testRestoreSnapshot-1676174318078=cc0e0b01138d4eae5df1e5641b484f45-027aff06c1054aaaa903715be8404fab already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/testRestoreSnapshot-1676174318078=cc0e0b01138d4eae5df1e5641b484f45-027aff06c1054aaaa903715be8404fab. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343726 currentMtime: 1676174344149 2023-02-12 03:59:11,067 INFO [HFileArchiver-10] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/testRestoreSnapshot-1676174318078=ecda59e01589f4f279b19a5420a1e608-25b90241f56e47b781d04ecdbb8f5bd5 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/testRestoreSnapshot-1676174318078=ecda59e01589f4f279b19a5420a1e608-25b90241f56e47b781d04ecdbb8f5bd5.1676174351060. 2023-02-12 03:59:11,067 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/testRestoreSnapshot-1676174318078=fcd2ed887eb9f47c07236428879982c1-2e835d2ef04e4f84a4c85680200dc844 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/testRestoreSnapshot-1676174318078=fcd2ed887eb9f47c07236428879982c1-2e835d2ef04e4f84a4c85680200dc844. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343724 currentMtime: 1676174344115 2023-02-12 03:59:11,068 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/testRestoreSnapshot-1676174318078=609b1a331ab507675f39b4bdbe7a382c-1029f052cfe34ddb8506a2f181ab2423 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/testRestoreSnapshot-1676174318078=609b1a331ab507675f39b4bdbe7a382c-1029f052cfe34ddb8506a2f181ab2423 2023-02-12 03:59:11,068 INFO [HFileArchiver-9] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/testRestoreSnapshot-1676174318078=cc0e0b01138d4eae5df1e5641b484f45-027aff06c1054aaaa903715be8404fab to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/testRestoreSnapshot-1676174318078=cc0e0b01138d4eae5df1e5641b484f45-027aff06c1054aaaa903715be8404fab.1676174351060. 2023-02-12 03:59:11,068 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/testRestoreSnapshot-1676174318078=b6894e9d4c505cdcd16894575e6b138a-6ae5fbf6b3ca4a899b9e29e73711608c to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/testRestoreSnapshot-1676174318078=b6894e9d4c505cdcd16894575e6b138a-6ae5fbf6b3ca4a899b9e29e73711608c 2023-02-12 03:59:11,068 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/testRestoreSnapshot-1676174318078=fcd2ed887eb9f47c07236428879982c1-2e835d2ef04e4f84a4c85680200dc844 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/testRestoreSnapshot-1676174318078=fcd2ed887eb9f47c07236428879982c1-2e835d2ef04e4f84a4c85680200dc844.1676174351060. 2023-02-12 03:59:11,069 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/testRestoreSnapshot-1676174318078=cc0e0b01138d4eae5df1e5641b484f45-027aff06c1054aaaa903715be8404fab to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/testRestoreSnapshot-1676174318078=cc0e0b01138d4eae5df1e5641b484f45-027aff06c1054aaaa903715be8404fab 2023-02-12 03:59:11,070 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/testRestoreSnapshot-1676174318078=fcd2ed887eb9f47c07236428879982c1-2e835d2ef04e4f84a4c85680200dc844 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/testRestoreSnapshot-1676174318078=fcd2ed887eb9f47c07236428879982c1-2e835d2ef04e4f84a4c85680200dc844 2023-02-12 03:59:11,070 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/testRestoreSnapshot-1676174318078=ecda59e01589f4f279b19a5420a1e608-25b90241f56e47b781d04ecdbb8f5bd5 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/testRestoreSnapshot-1676174318078=ecda59e01589f4f279b19a5420a1e608-25b90241f56e47b781d04ecdbb8f5bd5 2023-02-12 03:59:11,070 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/testRestoreSnapshot-1676174318078=e019e3f4c00b3da154d59b4085911d76-6148ef2edb69420e9208fff789b259f0 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/testRestoreSnapshot-1676174318078=e019e3f4c00b3da154d59b4085911d76-6148ef2edb69420e9208fff789b259f0. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343728 currentMtime: 1676174344156 2023-02-12 03:59:11,071 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/testRestoreSnapshot-1676174318078=e019e3f4c00b3da154d59b4085911d76-6148ef2edb69420e9208fff789b259f0 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/testRestoreSnapshot-1676174318078=e019e3f4c00b3da154d59b4085911d76-6148ef2edb69420e9208fff789b259f0.1676174351061. 2023-02-12 03:59:11,071 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/testRestoreSnapshot-1676174318078=b6894e9d4c505cdcd16894575e6b138a-edd4e077572a45bcac8218359e3d1f91 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/testRestoreSnapshot-1676174318078=b6894e9d4c505cdcd16894575e6b138a-edd4e077572a45bcac8218359e3d1f91. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343725 currentMtime: 1676174344153 2023-02-12 03:59:11,072 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/testRestoreSnapshot-1676174318078=b6894e9d4c505cdcd16894575e6b138a-edd4e077572a45bcac8218359e3d1f91 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/testRestoreSnapshot-1676174318078=b6894e9d4c505cdcd16894575e6b138a-edd4e077572a45bcac8218359e3d1f91.1676174351060. 2023-02-12 03:59:11,072 ERROR [HFileArchiver-10] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/testRestoreSnapshot-1676174318078=ecda59e01589f4f279b19a5420a1e608-f75b9c751c584507ac2da36d4b2eb053 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/testRestoreSnapshot-1676174318078=ecda59e01589f4f279b19a5420a1e608-f75b9c751c584507ac2da36d4b2eb053. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343728 currentMtime: 1676174344143 2023-02-12 03:59:11,072 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/testRestoreSnapshot-1676174318078=609b1a331ab507675f39b4bdbe7a382c-de3f19ac4a9247fbac10bbe6eed0d5d1 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/testRestoreSnapshot-1676174318078=609b1a331ab507675f39b4bdbe7a382c-de3f19ac4a9247fbac10bbe6eed0d5d1. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343727 currentMtime: 1676174344162 2023-02-12 03:59:11,072 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/testRestoreSnapshot-1676174318078=e019e3f4c00b3da154d59b4085911d76-6148ef2edb69420e9208fff789b259f0 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/cf/testRestoreSnapshot-1676174318078=e019e3f4c00b3da154d59b4085911d76-6148ef2edb69420e9208fff789b259f0 2023-02-12 03:59:11,073 ERROR [HFileArchiver-9] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/testRestoreSnapshot-1676174318078=cc0e0b01138d4eae5df1e5641b484f45-10e6966d5cc341688a6163ff0d1d3493 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/testRestoreSnapshot-1676174318078=cc0e0b01138d4eae5df1e5641b484f45-10e6966d5cc341688a6163ff0d1d3493. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343735 currentMtime: 1676174344154 2023-02-12 03:59:11,073 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/testRestoreSnapshot-1676174318078=609b1a331ab507675f39b4bdbe7a382c-de3f19ac4a9247fbac10bbe6eed0d5d1 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/testRestoreSnapshot-1676174318078=609b1a331ab507675f39b4bdbe7a382c-de3f19ac4a9247fbac10bbe6eed0d5d1.1676174351060. 2023-02-12 03:59:11,073 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/testRestoreSnapshot-1676174318078=fcd2ed887eb9f47c07236428879982c1-f73ec6d83ce444e98c68cfc9bb0662a3 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/testRestoreSnapshot-1676174318078=fcd2ed887eb9f47c07236428879982c1-f73ec6d83ce444e98c68cfc9bb0662a3. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343728 currentMtime: 1676174344135 2023-02-12 03:59:11,073 INFO [HFileArchiver-9] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/testRestoreSnapshot-1676174318078=cc0e0b01138d4eae5df1e5641b484f45-10e6966d5cc341688a6163ff0d1d3493 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/testRestoreSnapshot-1676174318078=cc0e0b01138d4eae5df1e5641b484f45-10e6966d5cc341688a6163ff0d1d3493.1676174351060. 2023-02-12 03:59:11,073 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/testRestoreSnapshot-1676174318078=b6894e9d4c505cdcd16894575e6b138a-edd4e077572a45bcac8218359e3d1f91 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/cf/testRestoreSnapshot-1676174318078=b6894e9d4c505cdcd16894575e6b138a-edd4e077572a45bcac8218359e3d1f91 2023-02-12 03:59:11,073 INFO [HFileArchiver-10] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/testRestoreSnapshot-1676174318078=ecda59e01589f4f279b19a5420a1e608-f75b9c751c584507ac2da36d4b2eb053 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/testRestoreSnapshot-1676174318078=ecda59e01589f4f279b19a5420a1e608-f75b9c751c584507ac2da36d4b2eb053.1676174351060. 2023-02-12 03:59:11,074 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/testRestoreSnapshot-1676174318078=fcd2ed887eb9f47c07236428879982c1-f73ec6d83ce444e98c68cfc9bb0662a3 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/testRestoreSnapshot-1676174318078=fcd2ed887eb9f47c07236428879982c1-f73ec6d83ce444e98c68cfc9bb0662a3.1676174351060. 2023-02-12 03:59:11,075 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/testRestoreSnapshot-1676174318078=ecda59e01589f4f279b19a5420a1e608-f75b9c751c584507ac2da36d4b2eb053 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/cf/testRestoreSnapshot-1676174318078=ecda59e01589f4f279b19a5420a1e608-f75b9c751c584507ac2da36d4b2eb053 2023-02-12 03:59:11,075 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/testRestoreSnapshot-1676174318078=cc0e0b01138d4eae5df1e5641b484f45-10e6966d5cc341688a6163ff0d1d3493 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/cf/testRestoreSnapshot-1676174318078=cc0e0b01138d4eae5df1e5641b484f45-10e6966d5cc341688a6163ff0d1d3493 2023-02-12 03:59:11,075 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/testRestoreSnapshot-1676174318078=609b1a331ab507675f39b4bdbe7a382c-de3f19ac4a9247fbac10bbe6eed0d5d1 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/cf/testRestoreSnapshot-1676174318078=609b1a331ab507675f39b4bdbe7a382c-de3f19ac4a9247fbac10bbe6eed0d5d1 2023-02-12 03:59:11,075 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/testRestoreSnapshot-1676174318078=fcd2ed887eb9f47c07236428879982c1-f73ec6d83ce444e98c68cfc9bb0662a3 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/cf/testRestoreSnapshot-1676174318078=fcd2ed887eb9f47c07236428879982c1-f73ec6d83ce444e98c68cfc9bb0662a3 2023-02-12 03:59:11,077 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/recovered.edits/18.seqid to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a/recovered.edits/18.seqid 2023-02-12 03:59:11,077 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/recovered.edits/18.seqid to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76/recovered.edits/18.seqid 2023-02-12 03:59:11,077 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/recovered.edits/18.seqid to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c/recovered.edits/18.seqid 2023-02-12 03:59:11,078 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/b6894e9d4c505cdcd16894575e6b138a 2023-02-12 03:59:11,079 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/609b1a331ab507675f39b4bdbe7a382c 2023-02-12 03:59:11,080 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/recovered.edits/18.seqid to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45/recovered.edits/18.seqid 2023-02-12 03:59:11,081 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/e019e3f4c00b3da154d59b4085911d76 2023-02-12 03:59:11,081 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/recovered.edits/18.seqid to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1/recovered.edits/18.seqid 2023-02-12 03:59:11,082 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/fcd2ed887eb9f47c07236428879982c1 2023-02-12 03:59:11,083 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/cc0e0b01138d4eae5df1e5641b484f45 2023-02-12 03:59:11,088 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/recovered.edits/18.seqid to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608/recovered.edits/18.seqid 2023-02-12 03:59:11,089 DEBUG [HFileArchiver-10] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testRestoreSnapshot-1676174318078/ecda59e01589f4f279b19a5420a1e608 2023-02-12 03:59:11,089 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676174318078 regions 2023-02-12 03:59:11,090 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349 2023-02-12 03:59:11,091 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf] 2023-02-12 03:59:11,095 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343743 currentMtime: 1676174344024 2023-02-12 03:59:11,096 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f.1676174351091. 2023-02-12 03:59:11,097 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-a87ff679a2f3e71d9181a67b7542122c20230212beb313e47a54497bb3d53e370c02906f 2023-02-12 03:59:11,099 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343745 currentMtime: 1676174344028 2023-02-12 03:59:11,099 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b.1676174351091. 2023-02-12 03:59:11,102 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-a87ff679a2f3e71d9181a67b7542122c20230212cb7f279d021b4807a033534fcfa6ab1b 2023-02-12 03:59:11,104 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343746 currentMtime: 1676174344033 2023-02-12 03:59:11,105 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4.1676174351091. 2023-02-12 03:59:11,105 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c4ca4238a0b923820dcc509a6f75849b20230212087ced4f31ec44d2899d13b628c03ae4 2023-02-12 03:59:11,106 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343746 currentMtime: 1676174344035 2023-02-12 03:59:11,107 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64.1676174351091. 2023-02-12 03:59:11,109 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c4ca4238a0b923820dcc509a6f75849b202302121bfdcf0a8fc9482190f447282a980d64 2023-02-12 03:59:11,112 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343747 currentMtime: 1676174344037 2023-02-12 03:59:11,112 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900.1676174351091. 2023-02-12 03:59:11,115 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c81e728d9d4c2f636f067f89cc14862c202302120fd88ab51ce34fb09ae3f0c6351e5900 2023-02-12 03:59:11,117 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343748 currentMtime: 1676174344039 2023-02-12 03:59:11,118 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c.1676174351091. 2023-02-12 03:59:11,119 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-c81e728d9d4c2f636f067f89cc14862c202302129f1a7867c2914e1385a0d974a4d3fd3c 2023-02-12 03:59:11,120 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343749 currentMtime: 1676174344041 2023-02-12 03:59:11,120 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b.1676174351091. 2023-02-12 03:59:11,121 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-d41d8cd98f00b204e9800998ecf8427e202302121259221d58604a18b927add85009761b 2023-02-12 03:59:11,122 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343750 currentMtime: 1676174344043 2023-02-12 03:59:11,123 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06.1676174351091. 2023-02-12 03:59:11,125 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-d41d8cd98f00b204e9800998ecf8427e20230212c4700e7a57584498bcc46ca362f45f06 2023-02-12 03:59:11,129 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343751 currentMtime: 1676174344045 2023-02-12 03:59:11,130 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411.1676174351091. 2023-02-12 03:59:11,131 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-e4da3b7fbbce2345d7772b0674a318d5202302122356b1909a8b49979b938cbd54cf0411 2023-02-12 03:59:11,134 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343753 currentMtime: 1676174344047 2023-02-12 03:59:11,135 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89.1676174351091. 2023-02-12 03:59:11,136 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-e4da3b7fbbce2345d7772b0674a318d520230212d85aa17209384c32972414c475d5aa89 2023-02-12 03:59:11,137 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343754 currentMtime: 1676174344049 2023-02-12 03:59:11,138 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2.1676174351091. 2023-02-12 03:59:11,139 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-eccbc87e4b5ce2fe28308fd9f2a7baf3202302123e4cd114d6b94c3aa49eede1022110a2 2023-02-12 03:59:11,140 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28. archiveLen: 0 currentLen: 0 archiveMtime: 1676174343756 currentMtime: 1676174344051 2023-02-12 03:59:11,140 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28.1676174351091. 2023-02-12 03:59:11,141 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349/cf/testRestoreSnapshot-1676174318078=c311163dd62fe758baed239605c12349-eccbc87e4b5ce2fe28308fd9f2a7baf320230212d957a4a8cab54e099de02f67d8c60d28 2023-02-12 03:59:11,141 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testRestoreSnapshot-1676174318078/c311163dd62fe758baed239605c12349 2023-02-12 03:59:11,145 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=201, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:11,153 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676174318078 from hbase:meta 2023-02-12 03:59:11,156 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676174318078' descriptor. 2023-02-12 03:59:11,158 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=201, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:11,158 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676174318078' from region states. 2023-02-12 03:59:11,158 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174351158"}]},"ts":"9223372036854775807"} 2023-02-12 03:59:11,158 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174351158"}]},"ts":"9223372036854775807"} 2023-02-12 03:59:11,158 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174351158"}]},"ts":"9223372036854775807"} 2023-02-12 03:59:11,158 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174351158"}]},"ts":"9223372036854775807"} 2023-02-12 03:59:11,158 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174351158"}]},"ts":"9223372036854775807"} 2023-02-12 03:59:11,158 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174351158"}]},"ts":"9223372036854775807"} 2023-02-12 03:59:11,170 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-12 03:59:11,170 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => cc0e0b01138d4eae5df1e5641b484f45, NAME => 'testRestoreSnapshot-1676174318078,,1676174318099.cc0e0b01138d4eae5df1e5641b484f45.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => ecda59e01589f4f279b19a5420a1e608, NAME => 'testRestoreSnapshot-1676174318078,1,1676174318099.ecda59e01589f4f279b19a5420a1e608.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => fcd2ed887eb9f47c07236428879982c1, NAME => 'testRestoreSnapshot-1676174318078,2,1676174318099.fcd2ed887eb9f47c07236428879982c1.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 609b1a331ab507675f39b4bdbe7a382c, NAME => 'testRestoreSnapshot-1676174318078,3,1676174318099.609b1a331ab507675f39b4bdbe7a382c.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => e019e3f4c00b3da154d59b4085911d76, NAME => 'testRestoreSnapshot-1676174318078,4,1676174318099.e019e3f4c00b3da154d59b4085911d76.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => b6894e9d4c505cdcd16894575e6b138a, NAME => 'testRestoreSnapshot-1676174318078,5,1676174318099.b6894e9d4c505cdcd16894575e6b138a.', STARTKEY => '5', ENDKEY => ''}] 2023-02-12 03:59:11,170 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676174318078' as deleted. 2023-02-12 03:59:11,170 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676174318078","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676174351170"}]},"ts":"9223372036854775807"} 2023-02-12 03:59:11,177 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676174318078 state from META 2023-02-12 03:59:11,179 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=201, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676174318078 2023-02-12 03:59:11,181 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=201, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676174318078 in 139 msec 2023-02-12 03:59:11,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-12 03:59:11,298 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676174318078, procId: 201 completed 2023-02-12 03:59:11,309 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(748): Client=jenkins//148.251.75.209 delete name: "emptySnaptb-1676174318078" 2023-02-12 03:59:11,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676174318078 2023-02-12 03:59:11,313 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(748): Client=jenkins//148.251.75.209 delete name: "snaptb0-1676174318078" 2023-02-12 03:59:11,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676174318078 2023-02-12 03:59:11,318 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(748): Client=jenkins//148.251.75.209 delete name: "snaptb1-1676174318078" 2023-02-12 03:59:11,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(333): Deleting snapshot: snaptb1-1676174318078 2023-02-12 03:59:11,351 INFO [Listener at localhost.localdomain/46483] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=408 (was 358) Potentially hanging thread: MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x51e75f37-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7a990b1c-shared-pool-27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:50368@0x79efc34c sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$45/1760800848.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-891324994_15 at /127.0.0.1:39648 [Waiting for operation #493] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-16 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-463782719_15 at /127.0.0.1:39640 [Waiting for operation #516] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:50368@0x79efc34c-SendThread(127.0.0.1:50368) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-463782719_15 at /127.0.0.1:47114 [Waiting for operation #479] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:50368@0x79efc34c-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7a990b1c-shared-pool-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x383008f9-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-463782719_15 at /127.0.0.1:47106 [Waiting for operation #515] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-463782719_15 at /127.0.0.1:47130 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/cluster_9ba13d3e-ed14-b1d2-0f74-8c456c2a06ab/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/cluster_9ba13d3e-ed14-b1d2-0f74-8c456c2a06ab/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-463782719_15 at /127.0.0.1:35210 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7a990b1c-shared-pool-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7a990b1c-shared-pool-29 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/cluster_9ba13d3e-ed14-b1d2-0f74-8c456c2a06ab/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/cluster_9ba13d3e-ed14-b1d2-0f74-8c456c2a06ab/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/cluster_9ba13d3e-ed14-b1d2-0f74-8c456c2a06ab/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/cluster_9ba13d3e-ed14-b1d2-0f74-8c456c2a06ab/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=786 (was 686) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=320 (was 143) - SystemLoadAverage LEAK? -, ProcessCount=164 (was 164), AvailableMemoryMB=3755 (was 5017) 2023-02-12 03:59:11,363 INFO [Listener at localhost.localdomain/46483] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=408, OpenFileDescriptor=786, MaxFileDescriptor=60000, SystemLoadAverage=320, ProcessCount=164, AvailableMemoryMB=3754 2023-02-12 03:59:11,366 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$4(2102): Client=jenkins//148.251.75.209 create 'testCorruptedSnapshot-1676174351364', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-12 03:59:11,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCorruptedSnapshot-1676174351364 2023-02-12 03:59:11,368 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676174351364 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-12 03:59:11,368 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "testCorruptedSnapshot-1676174351364" procId is: 202 2023-02-12 03:59:11,369 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676174351364 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-12 03:59:11,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-12 03:59:11,373 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:11,373 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:11,373 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:11,373 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:11,373 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:11,373 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040 2023-02-12 03:59:11,373 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b empty. 2023-02-12 03:59:11,373 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d empty. 2023-02-12 03:59:11,373 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89 empty. 2023-02-12 03:59:11,374 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040 empty. 2023-02-12 03:59:11,374 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460 empty. 2023-02-12 03:59:11,374 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d empty. 2023-02-12 03:59:11,374 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:11,374 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040 2023-02-12 03:59:11,374 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:11,374 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:11,374 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:11,374 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:11,374 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676174351364 regions 2023-02-12 03:59:11,388 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/.tabledesc/.tableinfo.0000000001 2023-02-12 03:59:11,389 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-2] regionserver.HRegion(7671): creating {ENCODED => 8d6f7fc2ff29d0e8082a3acf6b72d460, NAME => 'testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testCorruptedSnapshot-1676174351364', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp 2023-02-12 03:59:11,390 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-0] regionserver.HRegion(7671): creating {ENCODED => ac12d10f8881ef03c212b7559a1cb27b, NAME => 'testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testCorruptedSnapshot-1676174351364', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp 2023-02-12 03:59:11,390 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-1] regionserver.HRegion(7671): creating {ENCODED => 1991ab242a39a7de82ac275617721040, NAME => 'testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testCorruptedSnapshot-1676174351364', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp 2023-02-12 03:59:11,410 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:11,410 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-2] regionserver.HRegion(1603): Closing 8d6f7fc2ff29d0e8082a3acf6b72d460, disabling compactions & flushes 2023-02-12 03:59:11,410 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:11,410 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:11,410 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. after waiting 0 ms 2023-02-12 03:59:11,410 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:11,410 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:11,410 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-2] regionserver.HRegion(1557): Region close journal for 8d6f7fc2ff29d0e8082a3acf6b72d460: Waiting for close lock at 1676174351410Disabling compacts and flushes for region at 1676174351410Disabling writes for close at 1676174351410Writing region close event to WAL at 1676174351410Closed at 1676174351410 2023-02-12 03:59:11,411 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-2] regionserver.HRegion(7671): creating {ENCODED => 395304cb23b0d8ab23e3ba3dc259249d, NAME => 'testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testCorruptedSnapshot-1676174351364', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp 2023-02-12 03:59:11,412 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:11,412 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-1] regionserver.HRegion(1603): Closing 1991ab242a39a7de82ac275617721040, disabling compactions & flushes 2023-02-12 03:59:11,412 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:11,412 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:11,412 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. after waiting 0 ms 2023-02-12 03:59:11,412 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:11,412 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:11,412 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-1] regionserver.HRegion(1557): Region close journal for 1991ab242a39a7de82ac275617721040: Waiting for close lock at 1676174351412Disabling compacts and flushes for region at 1676174351412Disabling writes for close at 1676174351412Writing region close event to WAL at 1676174351412Closed at 1676174351412 2023-02-12 03:59:11,413 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-1] regionserver.HRegion(7671): creating {ENCODED => 5e163755ca31807434b0e014e67c9f89, NAME => 'testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testCorruptedSnapshot-1676174351364', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp 2023-02-12 03:59:11,414 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:11,414 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-0] regionserver.HRegion(1603): Closing ac12d10f8881ef03c212b7559a1cb27b, disabling compactions & flushes 2023-02-12 03:59:11,414 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:11,414 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:11,414 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. after waiting 0 ms 2023-02-12 03:59:11,414 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:11,414 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:11,414 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-0] regionserver.HRegion(1557): Region close journal for ac12d10f8881ef03c212b7559a1cb27b: Waiting for close lock at 1676174351414Disabling compacts and flushes for region at 1676174351414Disabling writes for close at 1676174351414Writing region close event to WAL at 1676174351414Closed at 1676174351414 2023-02-12 03:59:11,414 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-0] regionserver.HRegion(7671): creating {ENCODED => 35bdba3092d7a6729f7490917901687d, NAME => 'testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testCorruptedSnapshot-1676174351364', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp 2023-02-12 03:59:11,419 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:11,419 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-2] regionserver.HRegion(1603): Closing 395304cb23b0d8ab23e3ba3dc259249d, disabling compactions & flushes 2023-02-12 03:59:11,419 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:11,419 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:11,419 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. after waiting 0 ms 2023-02-12 03:59:11,419 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:11,420 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:11,420 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-2] regionserver.HRegion(1557): Region close journal for 395304cb23b0d8ab23e3ba3dc259249d: Waiting for close lock at 1676174351419Disabling compacts and flushes for region at 1676174351419Disabling writes for close at 1676174351419Writing region close event to WAL at 1676174351420 (+1 ms)Closed at 1676174351420 2023-02-12 03:59:11,423 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:11,423 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-1] regionserver.HRegion(1603): Closing 5e163755ca31807434b0e014e67c9f89, disabling compactions & flushes 2023-02-12 03:59:11,423 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:11,423 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:11,423 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. after waiting 0 ms 2023-02-12 03:59:11,423 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:11,423 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:11,423 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-1] regionserver.HRegion(1557): Region close journal for 5e163755ca31807434b0e014e67c9f89: Waiting for close lock at 1676174351423Disabling compacts and flushes for region at 1676174351423Disabling writes for close at 1676174351423Writing region close event to WAL at 1676174351423Closed at 1676174351423 2023-02-12 03:59:11,424 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:11,424 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-0] regionserver.HRegion(1603): Closing 35bdba3092d7a6729f7490917901687d, disabling compactions & flushes 2023-02-12 03:59:11,424 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:11,424 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:11,424 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. after waiting 0 ms 2023-02-12 03:59:11,424 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:11,424 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:11,424 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676174351364-pool-0] regionserver.HRegion(1557): Region close journal for 35bdba3092d7a6729f7490917901687d: Waiting for close lock at 1676174351424Disabling compacts and flushes for region at 1676174351424Disabling writes for close at 1676174351424Writing region close event to WAL at 1676174351424Closed at 1676174351424 2023-02-12 03:59:11,426 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676174351364 execute state=CREATE_TABLE_ADD_TO_META 2023-02-12 03:59:11,427 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174351427"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174351427"}]},"ts":"1676174351427"} 2023-02-12 03:59:11,427 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174351427"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174351427"}]},"ts":"1676174351427"} 2023-02-12 03:59:11,427 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174351427"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174351427"}]},"ts":"1676174351427"} 2023-02-12 03:59:11,427 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174351427"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174351427"}]},"ts":"1676174351427"} 2023-02-12 03:59:11,427 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174351427"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174351427"}]},"ts":"1676174351427"} 2023-02-12 03:59:11,427 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174351427"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174351427"}]},"ts":"1676174351427"} 2023-02-12 03:59:11,431 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-12 03:59:11,431 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676174351364 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-12 03:59:11,431 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676174351364","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174351431"}]},"ts":"1676174351431"} 2023-02-12 03:59:11,432 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676174351364, state=ENABLING in hbase:meta 2023-02-12 03:59:11,435 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-02-12 03:59:11,438 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-12 03:59:11,438 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-12 03:59:11,438 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-12 03:59:11,438 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-12 03:59:11,438 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=ac12d10f8881ef03c212b7559a1cb27b, ASSIGN}, {pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=1991ab242a39a7de82ac275617721040, ASSIGN}, {pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=8d6f7fc2ff29d0e8082a3acf6b72d460, ASSIGN}, {pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=395304cb23b0d8ab23e3ba3dc259249d, ASSIGN}, {pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=5e163755ca31807434b0e014e67c9f89, ASSIGN}, {pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=35bdba3092d7a6729f7490917901687d, ASSIGN}] 2023-02-12 03:59:11,440 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=35bdba3092d7a6729f7490917901687d, ASSIGN 2023-02-12 03:59:11,441 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=395304cb23b0d8ab23e3ba3dc259249d, ASSIGN 2023-02-12 03:59:11,441 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=5e163755ca31807434b0e014e67c9f89, ASSIGN 2023-02-12 03:59:11,441 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=8d6f7fc2ff29d0e8082a3acf6b72d460, ASSIGN 2023-02-12 03:59:11,441 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=1991ab242a39a7de82ac275617721040, ASSIGN 2023-02-12 03:59:11,442 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=395304cb23b0d8ab23e3ba3dc259249d, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,42205,1676174315413; forceNewPlan=false, retain=false 2023-02-12 03:59:11,442 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=35bdba3092d7a6729f7490917901687d, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,35785,1676174315481; forceNewPlan=false, retain=false 2023-02-12 03:59:11,442 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=8d6f7fc2ff29d0e8082a3acf6b72d460, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,42205,1676174315413; forceNewPlan=false, retain=false 2023-02-12 03:59:11,442 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=5e163755ca31807434b0e014e67c9f89, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,35783,1676174315451; forceNewPlan=false, retain=false 2023-02-12 03:59:11,442 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=1991ab242a39a7de82ac275617721040, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,35783,1676174315451; forceNewPlan=false, retain=false 2023-02-12 03:59:11,442 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=ac12d10f8881ef03c212b7559a1cb27b, ASSIGN 2023-02-12 03:59:11,443 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=ac12d10f8881ef03c212b7559a1cb27b, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,35785,1676174315481; forceNewPlan=false, retain=false 2023-02-12 03:59:11,592 INFO [jenkins-hbase20:37305] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-12 03:59:11,595 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=ac12d10f8881ef03c212b7559a1cb27b, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:11,595 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=5e163755ca31807434b0e014e67c9f89, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:11,596 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174351595"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174351595"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174351595"}]},"ts":"1676174351595"} 2023-02-12 03:59:11,595 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=8d6f7fc2ff29d0e8082a3acf6b72d460, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:11,595 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=395304cb23b0d8ab23e3ba3dc259249d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:11,595 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=35bdba3092d7a6729f7490917901687d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:11,596 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174351595"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174351595"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174351595"}]},"ts":"1676174351595"} 2023-02-12 03:59:11,596 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174351595"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174351595"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174351595"}]},"ts":"1676174351595"} 2023-02-12 03:59:11,596 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174351595"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174351595"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174351595"}]},"ts":"1676174351595"} 2023-02-12 03:59:11,596 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174351595"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174351595"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174351595"}]},"ts":"1676174351595"} 2023-02-12 03:59:11,599 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=209, ppid=203, state=RUNNABLE; OpenRegionProcedure ac12d10f8881ef03c212b7559a1cb27b, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:59:11,601 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=210, ppid=208, state=RUNNABLE; OpenRegionProcedure 35bdba3092d7a6729f7490917901687d, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:59:11,603 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=211, ppid=206, state=RUNNABLE; OpenRegionProcedure 395304cb23b0d8ab23e3ba3dc259249d, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:59:11,604 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=1991ab242a39a7de82ac275617721040, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:11,605 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174351604"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174351604"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174351604"}]},"ts":"1676174351604"} 2023-02-12 03:59:11,611 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=212, ppid=205, state=RUNNABLE; OpenRegionProcedure 8d6f7fc2ff29d0e8082a3acf6b72d460, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:59:11,611 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=213, ppid=207, state=RUNNABLE; OpenRegionProcedure 5e163755ca31807434b0e014e67c9f89, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:59:11,612 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=214, ppid=204, state=RUNNABLE; OpenRegionProcedure 1991ab242a39a7de82ac275617721040, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:59:11,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-12 03:59:11,754 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:11,754 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 35bdba3092d7a6729f7490917901687d, NAME => 'testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.', STARTKEY => '5', ENDKEY => ''} 2023-02-12 03:59:11,755 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676174351364 35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:11,755 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:11,755 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:11,755 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:11,756 INFO [StoreOpener-35bdba3092d7a6729f7490917901687d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:11,757 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:11,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 395304cb23b0d8ab23e3ba3dc259249d, NAME => 'testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.', STARTKEY => '3', ENDKEY => '4'} 2023-02-12 03:59:11,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676174351364 395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:11,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:11,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:11,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:11,758 DEBUG [StoreOpener-35bdba3092d7a6729f7490917901687d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/cf 2023-02-12 03:59:11,758 DEBUG [StoreOpener-35bdba3092d7a6729f7490917901687d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/cf 2023-02-12 03:59:11,759 INFO [StoreOpener-35bdba3092d7a6729f7490917901687d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 35bdba3092d7a6729f7490917901687d columnFamilyName cf 2023-02-12 03:59:11,759 INFO [StoreOpener-395304cb23b0d8ab23e3ba3dc259249d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:11,762 DEBUG [StoreOpener-395304cb23b0d8ab23e3ba3dc259249d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/cf 2023-02-12 03:59:11,762 INFO [StoreOpener-35bdba3092d7a6729f7490917901687d-1] regionserver.HStore(310): Store=35bdba3092d7a6729f7490917901687d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:11,762 DEBUG [StoreOpener-395304cb23b0d8ab23e3ba3dc259249d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/cf 2023-02-12 03:59:11,763 INFO [StoreOpener-395304cb23b0d8ab23e3ba3dc259249d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 395304cb23b0d8ab23e3ba3dc259249d columnFamilyName cf 2023-02-12 03:59:11,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:11,764 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:11,764 INFO [StoreOpener-395304cb23b0d8ab23e3ba3dc259249d-1] regionserver.HStore(310): Store=395304cb23b0d8ab23e3ba3dc259249d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:11,765 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:11,765 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:11,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:11,766 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:11,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1991ab242a39a7de82ac275617721040, NAME => 'testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.', STARTKEY => '1', ENDKEY => '2'} 2023-02-12 03:59:11,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676174351364 1991ab242a39a7de82ac275617721040 2023-02-12 03:59:11,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:11,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 1991ab242a39a7de82ac275617721040 2023-02-12 03:59:11,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 1991ab242a39a7de82ac275617721040 2023-02-12 03:59:11,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:11,769 INFO [StoreOpener-1991ab242a39a7de82ac275617721040-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1991ab242a39a7de82ac275617721040 2023-02-12 03:59:11,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 03:59:11,776 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 35bdba3092d7a6729f7490917901687d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70300022, jitterRate=0.0475519597530365}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:11,776 DEBUG [StoreOpener-1991ab242a39a7de82ac275617721040-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/cf 2023-02-12 03:59:11,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 35bdba3092d7a6729f7490917901687d: Running coprocessor pre-open hook at 1676174351755Writing region info on filesystem at 1676174351755Initializing all the Stores at 1676174351755Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174351755Cleaning up temporary data from old regions at 1676174351764 (+9 ms)Cleaning up detritus from prior splits at 1676174351764Running coprocessor post-open hooks at 1676174351776 (+12 ms)Region opened successfully at 1676174351776 2023-02-12 03:59:11,776 DEBUG [StoreOpener-1991ab242a39a7de82ac275617721040-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/cf 2023-02-12 03:59:11,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 03:59:11,777 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 395304cb23b0d8ab23e3ba3dc259249d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73974721, jitterRate=0.10230924189090729}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:11,777 INFO [StoreOpener-1991ab242a39a7de82ac275617721040-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1991ab242a39a7de82ac275617721040 columnFamilyName cf 2023-02-12 03:59:11,777 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d., pid=210, masterSystemTime=1676174351752 2023-02-12 03:59:11,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 395304cb23b0d8ab23e3ba3dc259249d: Running coprocessor pre-open hook at 1676174351758Writing region info on filesystem at 1676174351758Initializing all the Stores at 1676174351758Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174351758Cleaning up temporary data from old regions at 1676174351766 (+8 ms)Cleaning up detritus from prior splits at 1676174351766Running coprocessor post-open hooks at 1676174351777 (+11 ms)Region opened successfully at 1676174351777 2023-02-12 03:59:11,778 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d., pid=211, masterSystemTime=1676174351755 2023-02-12 03:59:11,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:11,779 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:11,779 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:11,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ac12d10f8881ef03c212b7559a1cb27b, NAME => 'testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.', STARTKEY => '', ENDKEY => '1'} 2023-02-12 03:59:11,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676174351364 ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:11,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:11,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:11,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:11,783 INFO [StoreOpener-1991ab242a39a7de82ac275617721040-1] regionserver.HStore(310): Store=1991ab242a39a7de82ac275617721040/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:11,783 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=35bdba3092d7a6729f7490917901687d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:11,784 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174351783"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174351783"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174351783"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174351783"}]},"ts":"1676174351783"} 2023-02-12 03:59:11,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:11,784 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:11,784 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:11,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 8d6f7fc2ff29d0e8082a3acf6b72d460, NAME => 'testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.', STARTKEY => '2', ENDKEY => '3'} 2023-02-12 03:59:11,784 INFO [StoreOpener-ac12d10f8881ef03c212b7559a1cb27b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:11,784 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=395304cb23b0d8ab23e3ba3dc259249d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:11,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676174351364 8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:11,785 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174351784"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174351784"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174351784"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174351784"}]},"ts":"1676174351784"} 2023-02-12 03:59:11,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040 2023-02-12 03:59:11,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:11,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:11,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:11,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040 2023-02-12 03:59:11,787 DEBUG [StoreOpener-ac12d10f8881ef03c212b7559a1cb27b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/cf 2023-02-12 03:59:11,787 DEBUG [StoreOpener-ac12d10f8881ef03c212b7559a1cb27b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/cf 2023-02-12 03:59:11,787 INFO [StoreOpener-ac12d10f8881ef03c212b7559a1cb27b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ac12d10f8881ef03c212b7559a1cb27b columnFamilyName cf 2023-02-12 03:59:11,788 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=210, resume processing ppid=208 2023-02-12 03:59:11,788 INFO [StoreOpener-8d6f7fc2ff29d0e8082a3acf6b72d460-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:11,788 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=210, ppid=208, state=SUCCESS; OpenRegionProcedure 35bdba3092d7a6729f7490917901687d, server=jenkins-hbase20.apache.org,35785,1676174315481 in 184 msec 2023-02-12 03:59:11,789 INFO [StoreOpener-ac12d10f8881ef03c212b7559a1cb27b-1] regionserver.HStore(310): Store=ac12d10f8881ef03c212b7559a1cb27b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:11,789 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=211, resume processing ppid=206 2023-02-12 03:59:11,789 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=211, ppid=206, state=SUCCESS; OpenRegionProcedure 395304cb23b0d8ab23e3ba3dc259249d, server=jenkins-hbase20.apache.org,42205,1676174315413 in 183 msec 2023-02-12 03:59:11,789 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 1991ab242a39a7de82ac275617721040 2023-02-12 03:59:11,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:11,790 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=208, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=35bdba3092d7a6729f7490917901687d, ASSIGN in 350 msec 2023-02-12 03:59:11,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:11,790 DEBUG [StoreOpener-8d6f7fc2ff29d0e8082a3acf6b72d460-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/cf 2023-02-12 03:59:11,790 DEBUG [StoreOpener-8d6f7fc2ff29d0e8082a3acf6b72d460-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/cf 2023-02-12 03:59:11,790 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=206, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=395304cb23b0d8ab23e3ba3dc259249d, ASSIGN in 351 msec 2023-02-12 03:59:11,791 INFO [StoreOpener-8d6f7fc2ff29d0e8082a3acf6b72d460-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8d6f7fc2ff29d0e8082a3acf6b72d460 columnFamilyName cf 2023-02-12 03:59:11,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 03:59:11,792 INFO [StoreOpener-8d6f7fc2ff29d0e8082a3acf6b72d460-1] regionserver.HStore(310): Store=8d6f7fc2ff29d0e8082a3acf6b72d460/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:11,792 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 1991ab242a39a7de82ac275617721040; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75466802, jitterRate=0.12454298138618469}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:11,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 1991ab242a39a7de82ac275617721040: Running coprocessor pre-open hook at 1676174351767Writing region info on filesystem at 1676174351767Initializing all the Stores at 1676174351767Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174351767Cleaning up temporary data from old regions at 1676174351786 (+19 ms)Cleaning up detritus from prior splits at 1676174351786Running coprocessor post-open hooks at 1676174351792 (+6 ms)Region opened successfully at 1676174351792 2023-02-12 03:59:11,793 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040., pid=214, masterSystemTime=1676174351764 2023-02-12 03:59:11,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:11,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:11,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:11,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:11,794 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:11,794 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:11,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5e163755ca31807434b0e014e67c9f89, NAME => 'testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.', STARTKEY => '4', ENDKEY => '5'} 2023-02-12 03:59:11,795 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=1991ab242a39a7de82ac275617721040, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:11,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676174351364 5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:11,795 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174351794"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174351794"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174351794"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174351794"}]},"ts":"1676174351794"} 2023-02-12 03:59:11,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:11,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:11,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:11,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 03:59:11,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:11,797 INFO [StoreOpener-5e163755ca31807434b0e014e67c9f89-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:11,798 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=214, resume processing ppid=204 2023-02-12 03:59:11,799 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=214, ppid=204, state=SUCCESS; OpenRegionProcedure 1991ab242a39a7de82ac275617721040, server=jenkins-hbase20.apache.org,35783,1676174315451 in 185 msec 2023-02-12 03:59:11,799 DEBUG [StoreOpener-5e163755ca31807434b0e014e67c9f89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/cf 2023-02-12 03:59:11,799 DEBUG [StoreOpener-5e163755ca31807434b0e014e67c9f89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/cf 2023-02-12 03:59:11,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 03:59:11,799 INFO [StoreOpener-5e163755ca31807434b0e014e67c9f89-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5e163755ca31807434b0e014e67c9f89 columnFamilyName cf 2023-02-12 03:59:11,799 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 8d6f7fc2ff29d0e8082a3acf6b72d460; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65427208, jitterRate=-0.025058627128601074}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:11,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 8d6f7fc2ff29d0e8082a3acf6b72d460: Running coprocessor pre-open hook at 1676174351785Writing region info on filesystem at 1676174351785Initializing all the Stores at 1676174351786 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174351786Cleaning up temporary data from old regions at 1676174351794 (+8 ms)Cleaning up detritus from prior splits at 1676174351794Running coprocessor post-open hooks at 1676174351800 (+6 ms)Region opened successfully at 1676174351800 2023-02-12 03:59:11,800 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened ac12d10f8881ef03c212b7559a1cb27b; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60409732, jitterRate=-0.09982484579086304}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:11,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for ac12d10f8881ef03c212b7559a1cb27b: Running coprocessor pre-open hook at 1676174351780Writing region info on filesystem at 1676174351780Initializing all the Stores at 1676174351784 (+4 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174351784Cleaning up temporary data from old regions at 1676174351791 (+7 ms)Cleaning up detritus from prior splits at 1676174351791Running coprocessor post-open hooks at 1676174351800 (+9 ms)Region opened successfully at 1676174351800 2023-02-12 03:59:11,801 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460., pid=212, masterSystemTime=1676174351755 2023-02-12 03:59:11,801 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=204, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=1991ab242a39a7de82ac275617721040, ASSIGN in 361 msec 2023-02-12 03:59:11,802 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:11,802 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:11,802 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b., pid=209, masterSystemTime=1676174351752 2023-02-12 03:59:11,803 INFO [StoreOpener-5e163755ca31807434b0e014e67c9f89-1] regionserver.HStore(310): Store=5e163755ca31807434b0e014e67c9f89/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:11,804 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=8d6f7fc2ff29d0e8082a3acf6b72d460, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:11,804 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174351804"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174351804"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174351804"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174351804"}]},"ts":"1676174351804"} 2023-02-12 03:59:11,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:11,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:11,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:11,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:11,806 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=ac12d10f8881ef03c212b7559a1cb27b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:11,807 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174351806"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174351806"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174351806"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174351806"}]},"ts":"1676174351806"} 2023-02-12 03:59:11,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:11,809 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=212, resume processing ppid=205 2023-02-12 03:59:11,809 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=212, ppid=205, state=SUCCESS; OpenRegionProcedure 8d6f7fc2ff29d0e8082a3acf6b72d460, server=jenkins-hbase20.apache.org,42205,1676174315413 in 194 msec 2023-02-12 03:59:11,810 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=209, resume processing ppid=203 2023-02-12 03:59:11,810 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=205, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=8d6f7fc2ff29d0e8082a3acf6b72d460, ASSIGN in 371 msec 2023-02-12 03:59:11,810 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=209, ppid=203, state=SUCCESS; OpenRegionProcedure ac12d10f8881ef03c212b7559a1cb27b, server=jenkins-hbase20.apache.org,35785,1676174315481 in 209 msec 2023-02-12 03:59:11,811 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=203, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=ac12d10f8881ef03c212b7559a1cb27b, ASSIGN in 372 msec 2023-02-12 03:59:11,811 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 03:59:11,812 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 5e163755ca31807434b0e014e67c9f89; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62807281, jitterRate=-0.06409858167171478}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:11,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 5e163755ca31807434b0e014e67c9f89: Running coprocessor pre-open hook at 1676174351795Writing region info on filesystem at 1676174351795Initializing all the Stores at 1676174351797 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174351797Cleaning up temporary data from old regions at 1676174351805 (+8 ms)Cleaning up detritus from prior splits at 1676174351805Running coprocessor post-open hooks at 1676174351812 (+7 ms)Region opened successfully at 1676174351812 2023-02-12 03:59:11,812 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89., pid=213, masterSystemTime=1676174351764 2023-02-12 03:59:11,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:11,814 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:11,814 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=5e163755ca31807434b0e014e67c9f89, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:11,814 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174351814"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174351814"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174351814"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174351814"}]},"ts":"1676174351814"} 2023-02-12 03:59:11,817 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=213, resume processing ppid=207 2023-02-12 03:59:11,817 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=213, ppid=207, state=SUCCESS; OpenRegionProcedure 5e163755ca31807434b0e014e67c9f89, server=jenkins-hbase20.apache.org,35783,1676174315451 in 204 msec 2023-02-12 03:59:11,819 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=207, resume processing ppid=202 2023-02-12 03:59:11,819 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=207, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=5e163755ca31807434b0e014e67c9f89, ASSIGN in 379 msec 2023-02-12 03:59:11,820 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676174351364 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-12 03:59:11,820 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676174351364","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174351820"}]},"ts":"1676174351820"} 2023-02-12 03:59:11,821 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676174351364, state=ENABLED in hbase:meta 2023-02-12 03:59:11,825 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676174351364 execute state=CREATE_TABLE_POST_OPERATION 2023-02-12 03:59:11,826 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=202, state=SUCCESS; CreateTableProcedure table=testCorruptedSnapshot-1676174351364 in 459 msec 2023-02-12 03:59:12,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-12 03:59:12,122 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testCorruptedSnapshot-1676174351364, procId: 202 completed 2023-02-12 03:59:12,124 DEBUG [Listener at localhost.localdomain/46483] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676174351364 2023-02-12 03:59:12,125 DEBUG [Listener at localhost.localdomain/46483] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:12,127 INFO [Listener at localhost.localdomain/46483] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-12 03:59:12,141 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676174351364 2023-02-12 03:59:12,141 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$11(2405): Client=jenkins//148.251.75.209 disable testCorruptedSnapshot-1676174351364 2023-02-12 03:59:12,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=215, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676174351364 2023-02-12 03:59:12,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-12 03:59:12,144 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676174351364","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174352144"}]},"ts":"1676174352144"} 2023-02-12 03:59:12,145 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676174351364, state=DISABLING in hbase:meta 2023-02-12 03:59:12,147 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676174351364 to state=DISABLING 2023-02-12 03:59:12,147 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=ac12d10f8881ef03c212b7559a1cb27b, UNASSIGN}, {pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=1991ab242a39a7de82ac275617721040, UNASSIGN}, {pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=8d6f7fc2ff29d0e8082a3acf6b72d460, UNASSIGN}, {pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=395304cb23b0d8ab23e3ba3dc259249d, UNASSIGN}, {pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=5e163755ca31807434b0e014e67c9f89, UNASSIGN}, {pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=35bdba3092d7a6729f7490917901687d, UNASSIGN}] 2023-02-12 03:59:12,149 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=5e163755ca31807434b0e014e67c9f89, UNASSIGN 2023-02-12 03:59:12,149 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=8d6f7fc2ff29d0e8082a3acf6b72d460, UNASSIGN 2023-02-12 03:59:12,149 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=35bdba3092d7a6729f7490917901687d, UNASSIGN 2023-02-12 03:59:12,149 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=395304cb23b0d8ab23e3ba3dc259249d, UNASSIGN 2023-02-12 03:59:12,149 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=1991ab242a39a7de82ac275617721040, UNASSIGN 2023-02-12 03:59:12,149 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=5e163755ca31807434b0e014e67c9f89, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:12,150 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174352149"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174352149"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174352149"}]},"ts":"1676174352149"} 2023-02-12 03:59:12,150 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=35bdba3092d7a6729f7490917901687d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:12,150 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=8d6f7fc2ff29d0e8082a3acf6b72d460, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:12,150 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174352150"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174352150"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174352150"}]},"ts":"1676174352150"} 2023-02-12 03:59:12,150 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174352150"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174352150"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174352150"}]},"ts":"1676174352150"} 2023-02-12 03:59:12,150 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=395304cb23b0d8ab23e3ba3dc259249d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:12,150 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=1991ab242a39a7de82ac275617721040, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:12,150 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174352150"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174352150"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174352150"}]},"ts":"1676174352150"} 2023-02-12 03:59:12,150 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174352150"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174352150"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174352150"}]},"ts":"1676174352150"} 2023-02-12 03:59:12,151 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=222, ppid=220, state=RUNNABLE; CloseRegionProcedure 5e163755ca31807434b0e014e67c9f89, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:59:12,151 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=223, ppid=221, state=RUNNABLE; CloseRegionProcedure 35bdba3092d7a6729f7490917901687d, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:59:12,152 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=224, ppid=218, state=RUNNABLE; CloseRegionProcedure 8d6f7fc2ff29d0e8082a3acf6b72d460, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:59:12,152 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=225, ppid=219, state=RUNNABLE; CloseRegionProcedure 395304cb23b0d8ab23e3ba3dc259249d, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:59:12,153 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=226, ppid=217, state=RUNNABLE; CloseRegionProcedure 1991ab242a39a7de82ac275617721040, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:59:12,153 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=ac12d10f8881ef03c212b7559a1cb27b, UNASSIGN 2023-02-12 03:59:12,154 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=ac12d10f8881ef03c212b7559a1cb27b, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:12,154 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174352154"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174352154"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174352154"}]},"ts":"1676174352154"} 2023-02-12 03:59:12,155 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=227, ppid=216, state=RUNNABLE; CloseRegionProcedure ac12d10f8881ef03c212b7559a1cb27b, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:59:12,304 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:12,304 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 5e163755ca31807434b0e014e67c9f89, disabling compactions & flushes 2023-02-12 03:59:12,305 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:12,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:12,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. after waiting 0 ms 2023-02-12 03:59:12,305 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:12,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:12,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 395304cb23b0d8ab23e3ba3dc259249d, disabling compactions & flushes 2023-02-12 03:59:12,305 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:12,306 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:12,306 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing ac12d10f8881ef03c212b7559a1cb27b, disabling compactions & flushes 2023-02-12 03:59:12,306 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:12,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. after waiting 0 ms 2023-02-12 03:59:12,306 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:12,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:12,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:12,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. after waiting 0 ms 2023-02-12 03:59:12,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:12,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 03:59:12,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 03:59:12,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 03:59:12,311 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:12,311 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 5e163755ca31807434b0e014e67c9f89: Waiting for close lock at 1676174352304Running coprocessor pre-close hooks at 1676174352304Disabling compacts and flushes for region at 1676174352304Disabling writes for close at 1676174352305 (+1 ms)Writing region close event to WAL at 1676174352306 (+1 ms)Running coprocessor post-close hooks at 1676174352311 (+5 ms)Closed at 1676174352311 2023-02-12 03:59:12,311 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:12,311 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 395304cb23b0d8ab23e3ba3dc259249d: Waiting for close lock at 1676174352305Running coprocessor pre-close hooks at 1676174352305Disabling compacts and flushes for region at 1676174352305Disabling writes for close at 1676174352307 (+2 ms)Writing region close event to WAL at 1676174352308 (+1 ms)Running coprocessor post-close hooks at 1676174352311 (+3 ms)Closed at 1676174352311 2023-02-12 03:59:12,311 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:12,311 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for ac12d10f8881ef03c212b7559a1cb27b: Waiting for close lock at 1676174352306Running coprocessor pre-close hooks at 1676174352306Disabling compacts and flushes for region at 1676174352306Disabling writes for close at 1676174352307 (+1 ms)Writing region close event to WAL at 1676174352308 (+1 ms)Running coprocessor post-close hooks at 1676174352311 (+3 ms)Closed at 1676174352311 2023-02-12 03:59:12,312 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:12,312 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 1991ab242a39a7de82ac275617721040 2023-02-12 03:59:12,312 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 1991ab242a39a7de82ac275617721040, disabling compactions & flushes 2023-02-12 03:59:12,312 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:12,312 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:12,313 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. after waiting 0 ms 2023-02-12 03:59:12,313 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:12,313 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=5e163755ca31807434b0e014e67c9f89, regionState=CLOSED 2023-02-12 03:59:12,313 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174352313"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174352313"}]},"ts":"1676174352313"} 2023-02-12 03:59:12,313 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:12,313 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:12,313 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 8d6f7fc2ff29d0e8082a3acf6b72d460, disabling compactions & flushes 2023-02-12 03:59:12,313 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:12,313 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:12,313 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. after waiting 0 ms 2023-02-12 03:59:12,313 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:12,313 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=395304cb23b0d8ab23e3ba3dc259249d, regionState=CLOSED 2023-02-12 03:59:12,313 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174352313"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174352313"}]},"ts":"1676174352313"} 2023-02-12 03:59:12,313 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:12,313 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:12,314 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 35bdba3092d7a6729f7490917901687d, disabling compactions & flushes 2023-02-12 03:59:12,314 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:12,314 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:12,314 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. after waiting 0 ms 2023-02-12 03:59:12,314 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:12,314 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=ac12d10f8881ef03c212b7559a1cb27b, regionState=CLOSED 2023-02-12 03:59:12,315 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174352314"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174352314"}]},"ts":"1676174352314"} 2023-02-12 03:59:12,316 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 03:59:12,316 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 03:59:12,316 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:12,316 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:12,317 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 8d6f7fc2ff29d0e8082a3acf6b72d460: Waiting for close lock at 1676174352313Running coprocessor pre-close hooks at 1676174352313Disabling compacts and flushes for region at 1676174352313Disabling writes for close at 1676174352313Writing region close event to WAL at 1676174352313Running coprocessor post-close hooks at 1676174352316 (+3 ms)Closed at 1676174352316 2023-02-12 03:59:12,317 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 1991ab242a39a7de82ac275617721040: Waiting for close lock at 1676174352312Running coprocessor pre-close hooks at 1676174352312Disabling compacts and flushes for region at 1676174352312Disabling writes for close at 1676174352313 (+1 ms)Writing region close event to WAL at 1676174352313Running coprocessor post-close hooks at 1676174352316 (+3 ms)Closed at 1676174352316 2023-02-12 03:59:12,317 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 03:59:12,317 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:12,317 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 35bdba3092d7a6729f7490917901687d: Waiting for close lock at 1676174352314Running coprocessor pre-close hooks at 1676174352314Disabling compacts and flushes for region at 1676174352314Disabling writes for close at 1676174352314Writing region close event to WAL at 1676174352314Running coprocessor post-close hooks at 1676174352317 (+3 ms)Closed at 1676174352317 2023-02-12 03:59:12,317 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=222, resume processing ppid=220 2023-02-12 03:59:12,317 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=222, ppid=220, state=SUCCESS; CloseRegionProcedure 5e163755ca31807434b0e014e67c9f89, server=jenkins-hbase20.apache.org,35783,1676174315451 in 163 msec 2023-02-12 03:59:12,319 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=225, resume processing ppid=219 2023-02-12 03:59:12,319 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=225, ppid=219, state=SUCCESS; CloseRegionProcedure 395304cb23b0d8ab23e3ba3dc259249d, server=jenkins-hbase20.apache.org,42205,1676174315413 in 163 msec 2023-02-12 03:59:12,319 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:12,319 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=220, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=5e163755ca31807434b0e014e67c9f89, UNASSIGN in 170 msec 2023-02-12 03:59:12,319 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=8d6f7fc2ff29d0e8082a3acf6b72d460, regionState=CLOSED 2023-02-12 03:59:12,319 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174352319"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174352319"}]},"ts":"1676174352319"} 2023-02-12 03:59:12,320 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 1991ab242a39a7de82ac275617721040 2023-02-12 03:59:12,320 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=227, resume processing ppid=216 2023-02-12 03:59:12,320 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=1991ab242a39a7de82ac275617721040, regionState=CLOSED 2023-02-12 03:59:12,320 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=227, ppid=216, state=SUCCESS; CloseRegionProcedure ac12d10f8881ef03c212b7559a1cb27b, server=jenkins-hbase20.apache.org,35785,1676174315481 in 162 msec 2023-02-12 03:59:12,320 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174352320"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174352320"}]},"ts":"1676174352320"} 2023-02-12 03:59:12,320 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=219, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=395304cb23b0d8ab23e3ba3dc259249d, UNASSIGN in 172 msec 2023-02-12 03:59:12,320 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:12,321 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=35bdba3092d7a6729f7490917901687d, regionState=CLOSED 2023-02-12 03:59:12,321 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174352321"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174352321"}]},"ts":"1676174352321"} 2023-02-12 03:59:12,321 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=216, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=ac12d10f8881ef03c212b7559a1cb27b, UNASSIGN in 173 msec 2023-02-12 03:59:12,326 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=224, resume processing ppid=218 2023-02-12 03:59:12,326 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=224, ppid=218, state=SUCCESS; CloseRegionProcedure 8d6f7fc2ff29d0e8082a3acf6b72d460, server=jenkins-hbase20.apache.org,42205,1676174315413 in 173 msec 2023-02-12 03:59:12,327 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=226, resume processing ppid=217 2023-02-12 03:59:12,327 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=226, ppid=217, state=SUCCESS; CloseRegionProcedure 1991ab242a39a7de82ac275617721040, server=jenkins-hbase20.apache.org,35783,1676174315451 in 172 msec 2023-02-12 03:59:12,327 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=218, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=8d6f7fc2ff29d0e8082a3acf6b72d460, UNASSIGN in 179 msec 2023-02-12 03:59:12,328 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=223, resume processing ppid=221 2023-02-12 03:59:12,328 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=217, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=1991ab242a39a7de82ac275617721040, UNASSIGN in 180 msec 2023-02-12 03:59:12,328 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=223, ppid=221, state=SUCCESS; CloseRegionProcedure 35bdba3092d7a6729f7490917901687d, server=jenkins-hbase20.apache.org,35785,1676174315481 in 175 msec 2023-02-12 03:59:12,329 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=221, resume processing ppid=215 2023-02-12 03:59:12,329 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=221, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=35bdba3092d7a6729f7490917901687d, UNASSIGN in 181 msec 2023-02-12 03:59:12,329 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676174351364","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174352329"}]},"ts":"1676174352329"} 2023-02-12 03:59:12,330 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676174351364, state=DISABLED in hbase:meta 2023-02-12 03:59:12,331 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676174351364 to state=DISABLED 2023-02-12 03:59:12,332 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=215, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676174351364 in 191 msec 2023-02-12 03:59:12,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-12 03:59:12,396 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676174351364, procId: 215 completed 2023-02-12 03:59:12,397 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1580): Client=jenkins//148.251.75.209 snapshot request for:{ ss=emptySnaptb-1676174351364 table=testCorruptedSnapshot-1676174351364 type=FLUSH ttl=0 } 2023-02-12 03:59:12,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676174352397 (current time:1676174352397). 2023-02-12 03:59:12,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-12 03:59:12,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] zookeeper.ReadOnlyZKClient(139): Connect 0x1a2874db to 127.0.0.1:50368 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 03:59:12,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e1727c4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 03:59:12,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:59:12,412 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60370, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:59:12,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a2874db to 127.0.0.1:50368 2023-02-12 03:59:12,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:59:12,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-12 03:59:12,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676174351364 table=testCorruptedSnapshot-1676174351364 type=FLUSH ttl=0 } 2023-02-12 03:59:12,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676174351364, type=EXCLUSIVE 2023-02-12 03:59:12,416 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676174351364, type=EXCLUSIVE 2023-02-12 03:59:12,417 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=228, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676174351364, type=EXCLUSIVE; timeout=600000, timestamp=1676174952417 2023-02-12 03:59:12,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676174351364 table=testCorruptedSnapshot-1676174351364 type=FLUSH ttl=0 } 2023-02-12 03:59:12,417 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676174351364 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676174351364 2023-02-12 03:59:12,418 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676174351364 table=testCorruptedSnapshot-1676174351364 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-12 03:59:12,418 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-12 03:59:12,434 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676174351364 table=testCorruptedSnapshot-1676174351364 type=DISABLED ttl=0 } 2023-02-12 03:59:12,439 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:59:12,439 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:59:12,439 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:59:12,439 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:59:12,440 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:59:12,440 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:59:12,440 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:59:12,440 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:59:12,440 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:59:12,440 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:59:12,440 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:59:12,440 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:59:12,440 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:59:12,440 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:59:12,440 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 03:59:12,441 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 03:59:12,441 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 03:59:12,442 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 03:59:12,442 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 03:59:12,442 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 03:59:12,445 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 03:59:12,468 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676174351364 table=testCorruptedSnapshot-1676174351364 type=DISABLED ttl=0 } as finished. 2023-02-12 03:59:12,469 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676174351364 2023-02-12 03:59:12,469 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/emptySnaptb-1676174351364 2023-02-12 03:59:12,482 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/emptySnaptb-1676174351364 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/emptySnaptb-1676174351364 2023-02-12 03:59:12,503 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676174351364 of table testCorruptedSnapshot-1676174351364 completed 2023-02-12 03:59:12,503 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/emptySnaptb-1676174351364 2023-02-12 03:59:12,503 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/emptySnaptb-1676174351364 2023-02-12 03:59:12,503 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676174351364 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676174351364 at 1676174352417Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676174351364 table=testCorruptedSnapshot-1676174351364 type=DISABLED ttl=0 } at 1676174352434 (+17 ms)Consolidate snapshot: emptySnaptb-1676174351364 at 1676174352469 (+35 ms)Loading Region manifests for emptySnaptb-1676174351364 at 1676174352469Writing data manifest for emptySnaptb-1676174351364 at 1676174352471 (+2 ms)Verifying snapshot: emptySnaptb-1676174351364 at 1676174352478 (+7 ms)Snapshot emptySnaptb-1676174351364 of table testCorruptedSnapshot-1676174351364 completed at 1676174352503 (+25 ms) 2023-02-12 03:59:12,505 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=228, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676174351364, type=EXCLUSIVE 2023-02-12 03:59:12,506 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=228, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676174351364, type=EXCLUSIVE in 90 msec 2023-02-12 03:59:12,668 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 03:59:12,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676174351364 table=testCorruptedSnapshot-1676174351364 type=FLUSH ttl=0 } is done 2023-02-12 03:59:12,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676174351364 table=testCorruptedSnapshot-1676174351364 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-12 03:59:12,670 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676174351364 2023-02-12 03:59:12,670 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$10(2372): Client=jenkins//148.251.75.209 enable testCorruptedSnapshot-1676174351364 2023-02-12 03:59:12,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=229, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676174351364 2023-02-12 03:59:12,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-12 03:59:12,674 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676174351364 2023-02-12 03:59:12,674 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676174351364","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174352674"}]},"ts":"1676174352674"} 2023-02-12 03:59:12,675 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676174351364, state=ENABLING in hbase:meta 2023-02-12 03:59:12,677 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-12 03:59:12,677 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=ac12d10f8881ef03c212b7559a1cb27b, ASSIGN}, {pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=1991ab242a39a7de82ac275617721040, ASSIGN}, {pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=8d6f7fc2ff29d0e8082a3acf6b72d460, ASSIGN}, {pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=395304cb23b0d8ab23e3ba3dc259249d, ASSIGN}, {pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=5e163755ca31807434b0e014e67c9f89, ASSIGN}, {pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=35bdba3092d7a6729f7490917901687d, ASSIGN}] 2023-02-12 03:59:12,679 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=1991ab242a39a7de82ac275617721040, ASSIGN 2023-02-12 03:59:12,680 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=ac12d10f8881ef03c212b7559a1cb27b, ASSIGN 2023-02-12 03:59:12,680 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=8d6f7fc2ff29d0e8082a3acf6b72d460, ASSIGN 2023-02-12 03:59:12,680 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:12,680 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=35bdba3092d7a6729f7490917901687d, ASSIGN 2023-02-12 03:59:12,680 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=1991ab242a39a7de82ac275617721040, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,35783,1676174315451; forceNewPlan=false, retain=true 2023-02-12 03:59:12,682 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:12,682 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:12,682 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=5e163755ca31807434b0e014e67c9f89, ASSIGN 2023-02-12 03:59:12,682 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=8d6f7fc2ff29d0e8082a3acf6b72d460, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,42205,1676174315413; forceNewPlan=false, retain=true 2023-02-12 03:59:12,682 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=395304cb23b0d8ab23e3ba3dc259249d, ASSIGN 2023-02-12 03:59:12,682 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:12,682 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=ac12d10f8881ef03c212b7559a1cb27b, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,35785,1676174315481; forceNewPlan=false, retain=true 2023-02-12 03:59:12,682 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=35bdba3092d7a6729f7490917901687d, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,35785,1676174315481; forceNewPlan=false, retain=true 2023-02-12 03:59:12,683 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:12,683 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:12,683 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=5e163755ca31807434b0e014e67c9f89, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,35783,1676174315451; forceNewPlan=false, retain=true 2023-02-12 03:59:12,683 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=395304cb23b0d8ab23e3ba3dc259249d, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,42205,1676174315413; forceNewPlan=false, retain=true 2023-02-12 03:59:12,831 INFO [jenkins-hbase20:37305] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-12 03:59:12,831 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=395304cb23b0d8ab23e3ba3dc259249d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:12,831 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=35bdba3092d7a6729f7490917901687d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:12,831 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=5e163755ca31807434b0e014e67c9f89, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:12,832 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174352831"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174352831"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174352831"}]},"ts":"1676174352831"} 2023-02-12 03:59:12,831 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=8d6f7fc2ff29d0e8082a3acf6b72d460, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:12,831 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=ac12d10f8881ef03c212b7559a1cb27b, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:12,832 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174352831"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174352831"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174352831"}]},"ts":"1676174352831"} 2023-02-12 03:59:12,832 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174352831"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174352831"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174352831"}]},"ts":"1676174352831"} 2023-02-12 03:59:12,832 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174352831"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174352831"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174352831"}]},"ts":"1676174352831"} 2023-02-12 03:59:12,832 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174352831"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174352831"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174352831"}]},"ts":"1676174352831"} 2023-02-12 03:59:12,834 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=236, ppid=235, state=RUNNABLE; OpenRegionProcedure 35bdba3092d7a6729f7490917901687d, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:59:12,835 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=237, ppid=232, state=RUNNABLE; OpenRegionProcedure 8d6f7fc2ff29d0e8082a3acf6b72d460, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:59:12,837 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=238, ppid=230, state=RUNNABLE; OpenRegionProcedure ac12d10f8881ef03c212b7559a1cb27b, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:59:12,838 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=239, ppid=234, state=RUNNABLE; OpenRegionProcedure 5e163755ca31807434b0e014e67c9f89, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:59:12,839 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=240, ppid=233, state=RUNNABLE; OpenRegionProcedure 395304cb23b0d8ab23e3ba3dc259249d, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:59:12,840 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=1991ab242a39a7de82ac275617721040, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:12,840 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174352840"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174352840"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174352840"}]},"ts":"1676174352840"} 2023-02-12 03:59:12,841 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=241, ppid=231, state=RUNNABLE; OpenRegionProcedure 1991ab242a39a7de82ac275617721040, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:59:12,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-12 03:59:12,991 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:12,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 395304cb23b0d8ab23e3ba3dc259249d, NAME => 'testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.', STARTKEY => '3', ENDKEY => '4'} 2023-02-12 03:59:12,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676174351364 395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:12,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:12,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:12,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:12,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:12,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 35bdba3092d7a6729f7490917901687d, NAME => 'testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.', STARTKEY => '5', ENDKEY => ''} 2023-02-12 03:59:12,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676174351364 35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:12,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:12,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:12,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:12,994 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:12,994 INFO [StoreOpener-395304cb23b0d8ab23e3ba3dc259249d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:12,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1991ab242a39a7de82ac275617721040, NAME => 'testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.', STARTKEY => '1', ENDKEY => '2'} 2023-02-12 03:59:12,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676174351364 1991ab242a39a7de82ac275617721040 2023-02-12 03:59:12,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:12,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 1991ab242a39a7de82ac275617721040 2023-02-12 03:59:12,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 1991ab242a39a7de82ac275617721040 2023-02-12 03:59:12,995 DEBUG [StoreOpener-395304cb23b0d8ab23e3ba3dc259249d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/cf 2023-02-12 03:59:12,995 DEBUG [StoreOpener-395304cb23b0d8ab23e3ba3dc259249d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/cf 2023-02-12 03:59:12,996 INFO [StoreOpener-395304cb23b0d8ab23e3ba3dc259249d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 395304cb23b0d8ab23e3ba3dc259249d columnFamilyName cf 2023-02-12 03:59:12,996 INFO [StoreOpener-395304cb23b0d8ab23e3ba3dc259249d-1] regionserver.HStore(310): Store=395304cb23b0d8ab23e3ba3dc259249d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:12,999 INFO [StoreOpener-35bdba3092d7a6729f7490917901687d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:12,999 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:13,001 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:13,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:13,005 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 395304cb23b0d8ab23e3ba3dc259249d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61148945, jitterRate=-0.08880971372127533}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:13,006 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 395304cb23b0d8ab23e3ba3dc259249d: Running coprocessor pre-open hook at 1676174352992Writing region info on filesystem at 1676174352992Initializing all the Stores at 1676174352993 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174352993Cleaning up temporary data from old regions at 1676174353002 (+9 ms)Cleaning up detritus from prior splits at 1676174353002Running coprocessor post-open hooks at 1676174353006 (+4 ms)Region opened successfully at 1676174353006 2023-02-12 03:59:13,006 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d., pid=240, masterSystemTime=1676174352988 2023-02-12 03:59:13,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:13,011 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:13,011 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:13,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 8d6f7fc2ff29d0e8082a3acf6b72d460, NAME => 'testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.', STARTKEY => '2', ENDKEY => '3'} 2023-02-12 03:59:13,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676174351364 8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:13,012 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=395304cb23b0d8ab23e3ba3dc259249d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:13,012 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174353012"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174353012"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174353012"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174353012"}]},"ts":"1676174353012"} 2023-02-12 03:59:13,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:13,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:13,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:13,019 INFO [StoreOpener-1991ab242a39a7de82ac275617721040-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1991ab242a39a7de82ac275617721040 2023-02-12 03:59:13,019 DEBUG [StoreOpener-35bdba3092d7a6729f7490917901687d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/cf 2023-02-12 03:59:13,019 DEBUG [StoreOpener-35bdba3092d7a6729f7490917901687d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/cf 2023-02-12 03:59:13,020 INFO [StoreOpener-35bdba3092d7a6729f7490917901687d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 35bdba3092d7a6729f7490917901687d columnFamilyName cf 2023-02-12 03:59:13,021 INFO [StoreOpener-35bdba3092d7a6729f7490917901687d-1] regionserver.HStore(310): Store=35bdba3092d7a6729f7490917901687d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:13,023 INFO [StoreOpener-8d6f7fc2ff29d0e8082a3acf6b72d460-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:13,024 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:13,024 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=240, resume processing ppid=233 2023-02-12 03:59:13,024 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=240, ppid=233, state=SUCCESS; OpenRegionProcedure 395304cb23b0d8ab23e3ba3dc259249d, server=jenkins-hbase20.apache.org,42205,1676174315413 in 175 msec 2023-02-12 03:59:13,025 DEBUG [StoreOpener-8d6f7fc2ff29d0e8082a3acf6b72d460-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/cf 2023-02-12 03:59:13,025 DEBUG [StoreOpener-8d6f7fc2ff29d0e8082a3acf6b72d460-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/cf 2023-02-12 03:59:13,025 DEBUG [StoreOpener-1991ab242a39a7de82ac275617721040-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/cf 2023-02-12 03:59:13,025 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:13,025 DEBUG [StoreOpener-1991ab242a39a7de82ac275617721040-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/cf 2023-02-12 03:59:13,026 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=233, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=395304cb23b0d8ab23e3ba3dc259249d, ASSIGN in 347 msec 2023-02-12 03:59:13,025 INFO [StoreOpener-8d6f7fc2ff29d0e8082a3acf6b72d460-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8d6f7fc2ff29d0e8082a3acf6b72d460 columnFamilyName cf 2023-02-12 03:59:13,027 INFO [StoreOpener-8d6f7fc2ff29d0e8082a3acf6b72d460-1] regionserver.HStore(310): Store=8d6f7fc2ff29d0e8082a3acf6b72d460/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:13,028 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:13,028 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:13,029 INFO [StoreOpener-1991ab242a39a7de82ac275617721040-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1991ab242a39a7de82ac275617721040 columnFamilyName cf 2023-02-12 03:59:13,029 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 35bdba3092d7a6729f7490917901687d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66362991, jitterRate=-0.011114373803138733}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:13,029 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 35bdba3092d7a6729f7490917901687d: Running coprocessor pre-open hook at 1676174352993Writing region info on filesystem at 1676174352993Initializing all the Stores at 1676174352994 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174352994Cleaning up temporary data from old regions at 1676174353026 (+32 ms)Cleaning up detritus from prior splits at 1676174353026Running coprocessor post-open hooks at 1676174353029 (+3 ms)Region opened successfully at 1676174353029 2023-02-12 03:59:13,029 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:13,030 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d., pid=236, masterSystemTime=1676174352987 2023-02-12 03:59:13,031 INFO [StoreOpener-1991ab242a39a7de82ac275617721040-1] regionserver.HStore(310): Store=1991ab242a39a7de82ac275617721040/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:13,032 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040 2023-02-12 03:59:13,033 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=35bdba3092d7a6729f7490917901687d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:13,033 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174353033"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174353033"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174353033"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174353033"}]},"ts":"1676174353033"} 2023-02-12 03:59:13,033 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040 2023-02-12 03:59:13,034 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:13,034 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:13,034 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:13,034 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:13,034 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ac12d10f8881ef03c212b7559a1cb27b, NAME => 'testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.', STARTKEY => '', ENDKEY => '1'} 2023-02-12 03:59:13,035 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676174351364 ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:13,035 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:13,035 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:13,035 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:13,036 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 8d6f7fc2ff29d0e8082a3acf6b72d460; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64382243, jitterRate=-0.04062981903553009}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:13,036 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 8d6f7fc2ff29d0e8082a3acf6b72d460: Running coprocessor pre-open hook at 1676174353013Writing region info on filesystem at 1676174353013Initializing all the Stores at 1676174353014 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174353014Cleaning up temporary data from old regions at 1676174353030 (+16 ms)Cleaning up detritus from prior splits at 1676174353030Running coprocessor post-open hooks at 1676174353036 (+6 ms)Region opened successfully at 1676174353036 2023-02-12 03:59:13,037 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460., pid=237, masterSystemTime=1676174352988 2023-02-12 03:59:13,038 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=236, resume processing ppid=235 2023-02-12 03:59:13,038 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=236, ppid=235, state=SUCCESS; OpenRegionProcedure 35bdba3092d7a6729f7490917901687d, server=jenkins-hbase20.apache.org,35785,1676174315481 in 200 msec 2023-02-12 03:59:13,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 1991ab242a39a7de82ac275617721040 2023-02-12 03:59:13,039 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 1991ab242a39a7de82ac275617721040; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71452888, jitterRate=0.06473100185394287}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:13,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 1991ab242a39a7de82ac275617721040: Running coprocessor pre-open hook at 1676174352994Writing region info on filesystem at 1676174352994Initializing all the Stores at 1676174352995 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174352995Cleaning up temporary data from old regions at 1676174353034 (+39 ms)Cleaning up detritus from prior splits at 1676174353035 (+1 ms)Running coprocessor post-open hooks at 1676174353039 (+4 ms)Region opened successfully at 1676174353039 2023-02-12 03:59:13,040 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=235, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=35bdba3092d7a6729f7490917901687d, ASSIGN in 361 msec 2023-02-12 03:59:13,040 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:13,040 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:13,040 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040., pid=241, masterSystemTime=1676174352990 2023-02-12 03:59:13,040 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=8d6f7fc2ff29d0e8082a3acf6b72d460, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:13,041 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174353040"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174353040"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174353040"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174353040"}]},"ts":"1676174353040"} 2023-02-12 03:59:13,043 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:13,043 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:13,043 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:13,043 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5e163755ca31807434b0e014e67c9f89, NAME => 'testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.', STARTKEY => '4', ENDKEY => '5'} 2023-02-12 03:59:13,043 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676174351364 5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:13,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:13,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:13,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:13,044 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=1991ab242a39a7de82ac275617721040, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:13,045 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174353044"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174353044"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174353044"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174353044"}]},"ts":"1676174353044"} 2023-02-12 03:59:13,048 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=237, resume processing ppid=232 2023-02-12 03:59:13,049 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=237, ppid=232, state=SUCCESS; OpenRegionProcedure 8d6f7fc2ff29d0e8082a3acf6b72d460, server=jenkins-hbase20.apache.org,42205,1676174315413 in 207 msec 2023-02-12 03:59:13,049 INFO [StoreOpener-ac12d10f8881ef03c212b7559a1cb27b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:13,051 INFO [StoreOpener-5e163755ca31807434b0e014e67c9f89-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:13,051 DEBUG [StoreOpener-ac12d10f8881ef03c212b7559a1cb27b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/cf 2023-02-12 03:59:13,051 DEBUG [StoreOpener-ac12d10f8881ef03c212b7559a1cb27b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/cf 2023-02-12 03:59:13,052 INFO [StoreOpener-ac12d10f8881ef03c212b7559a1cb27b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ac12d10f8881ef03c212b7559a1cb27b columnFamilyName cf 2023-02-12 03:59:13,052 DEBUG [StoreOpener-5e163755ca31807434b0e014e67c9f89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/cf 2023-02-12 03:59:13,052 DEBUG [StoreOpener-5e163755ca31807434b0e014e67c9f89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/cf 2023-02-12 03:59:13,052 INFO [StoreOpener-ac12d10f8881ef03c212b7559a1cb27b-1] regionserver.HStore(310): Store=ac12d10f8881ef03c212b7559a1cb27b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:13,053 INFO [StoreOpener-5e163755ca31807434b0e014e67c9f89-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5e163755ca31807434b0e014e67c9f89 columnFamilyName cf 2023-02-12 03:59:13,053 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:13,054 INFO [StoreOpener-5e163755ca31807434b0e014e67c9f89-1] regionserver.HStore(310): Store=5e163755ca31807434b0e014e67c9f89/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:13,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:13,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:13,055 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=232, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=8d6f7fc2ff29d0e8082a3acf6b72d460, ASSIGN in 371 msec 2023-02-12 03:59:13,056 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=241, resume processing ppid=231 2023-02-12 03:59:13,056 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=241, ppid=231, state=SUCCESS; OpenRegionProcedure 1991ab242a39a7de82ac275617721040, server=jenkins-hbase20.apache.org,35783,1676174315451 in 205 msec 2023-02-12 03:59:13,056 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:13,057 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:13,057 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=231, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=1991ab242a39a7de82ac275617721040, ASSIGN in 379 msec 2023-02-12 03:59:13,058 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened ac12d10f8881ef03c212b7559a1cb27b; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73650471, jitterRate=0.09747754037380219}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:13,058 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for ac12d10f8881ef03c212b7559a1cb27b: Running coprocessor pre-open hook at 1676174353035Writing region info on filesystem at 1676174353035Initializing all the Stores at 1676174353038 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174353038Cleaning up temporary data from old regions at 1676174353055 (+17 ms)Cleaning up detritus from prior splits at 1676174353055Running coprocessor post-open hooks at 1676174353058 (+3 ms)Region opened successfully at 1676174353058 2023-02-12 03:59:13,059 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b., pid=238, masterSystemTime=1676174352987 2023-02-12 03:59:13,059 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:13,060 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 5e163755ca31807434b0e014e67c9f89; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72407157, jitterRate=0.07895071804523468}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:13,060 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 5e163755ca31807434b0e014e67c9f89: Running coprocessor pre-open hook at 1676174353044Writing region info on filesystem at 1676174353044Initializing all the Stores at 1676174353049 (+5 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174353049Cleaning up temporary data from old regions at 1676174353057 (+8 ms)Cleaning up detritus from prior splits at 1676174353057Running coprocessor post-open hooks at 1676174353060 (+3 ms)Region opened successfully at 1676174353060 2023-02-12 03:59:13,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:13,061 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:13,061 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89., pid=239, masterSystemTime=1676174352990 2023-02-12 03:59:13,061 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=ac12d10f8881ef03c212b7559a1cb27b, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:13,061 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174353061"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174353061"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174353061"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174353061"}]},"ts":"1676174353061"} 2023-02-12 03:59:13,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:13,062 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:13,062 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=5e163755ca31807434b0e014e67c9f89, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:13,063 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174353062"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174353062"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174353062"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174353062"}]},"ts":"1676174353062"} 2023-02-12 03:59:13,063 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=238, resume processing ppid=230 2023-02-12 03:59:13,064 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=238, ppid=230, state=SUCCESS; OpenRegionProcedure ac12d10f8881ef03c212b7559a1cb27b, server=jenkins-hbase20.apache.org,35785,1676174315481 in 225 msec 2023-02-12 03:59:13,064 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=230, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=ac12d10f8881ef03c212b7559a1cb27b, ASSIGN in 387 msec 2023-02-12 03:59:13,065 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=239, resume processing ppid=234 2023-02-12 03:59:13,065 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=239, ppid=234, state=SUCCESS; OpenRegionProcedure 5e163755ca31807434b0e014e67c9f89, server=jenkins-hbase20.apache.org,35783,1676174315451 in 226 msec 2023-02-12 03:59:13,066 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=234, resume processing ppid=229 2023-02-12 03:59:13,066 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=234, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=5e163755ca31807434b0e014e67c9f89, ASSIGN in 388 msec 2023-02-12 03:59:13,067 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676174351364","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174353067"}]},"ts":"1676174353067"} 2023-02-12 03:59:13,068 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676174351364, state=ENABLED in hbase:meta 2023-02-12 03:59:13,069 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676174351364' was successfully enabled. 2023-02-12 03:59:13,070 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=229, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676174351364 in 399 msec 2023-02-12 03:59:13,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-12 03:59:13,426 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676174351364, procId: 229 completed 2023-02-12 03:59:13,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:59:13,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:59:13,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:59:13,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:59:13,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:59:13,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:59:13,438 DEBUG [Listener at localhost.localdomain/46483] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676174351364 2023-02-12 03:59:13,439 DEBUG [Listener at localhost.localdomain/46483] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:13,439 INFO [Listener at localhost.localdomain/46483] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-12 03:59:13,452 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-12 03:59:13,454 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-12 03:59:13,455 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-12 03:59:13,456 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-12 03:59:13,457 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-12 03:59:13,461 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676174351364 2023-02-12 03:59:13,462 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$11(2405): Client=jenkins//148.251.75.209 disable testCorruptedSnapshot-1676174351364 2023-02-12 03:59:13,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=242, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676174351364 2023-02-12 03:59:13,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-12 03:59:13,465 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676174351364","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174353465"}]},"ts":"1676174353465"} 2023-02-12 03:59:13,466 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676174351364, state=DISABLING in hbase:meta 2023-02-12 03:59:13,467 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676174351364 to state=DISABLING 2023-02-12 03:59:13,468 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=ac12d10f8881ef03c212b7559a1cb27b, UNASSIGN}, {pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=1991ab242a39a7de82ac275617721040, UNASSIGN}, {pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=8d6f7fc2ff29d0e8082a3acf6b72d460, UNASSIGN}, {pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=395304cb23b0d8ab23e3ba3dc259249d, UNASSIGN}, {pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=5e163755ca31807434b0e014e67c9f89, UNASSIGN}, {pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=35bdba3092d7a6729f7490917901687d, UNASSIGN}] 2023-02-12 03:59:13,470 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=5e163755ca31807434b0e014e67c9f89, UNASSIGN 2023-02-12 03:59:13,470 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=395304cb23b0d8ab23e3ba3dc259249d, UNASSIGN 2023-02-12 03:59:13,470 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=35bdba3092d7a6729f7490917901687d, UNASSIGN 2023-02-12 03:59:13,470 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=8d6f7fc2ff29d0e8082a3acf6b72d460, UNASSIGN 2023-02-12 03:59:13,470 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=1991ab242a39a7de82ac275617721040, UNASSIGN 2023-02-12 03:59:13,471 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=5e163755ca31807434b0e014e67c9f89, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:13,471 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=35bdba3092d7a6729f7490917901687d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:13,471 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174353471"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174353471"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174353471"}]},"ts":"1676174353471"} 2023-02-12 03:59:13,471 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=1991ab242a39a7de82ac275617721040, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:13,471 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174353471"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174353471"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174353471"}]},"ts":"1676174353471"} 2023-02-12 03:59:13,471 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=395304cb23b0d8ab23e3ba3dc259249d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:13,471 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174353471"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174353471"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174353471"}]},"ts":"1676174353471"} 2023-02-12 03:59:13,471 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=8d6f7fc2ff29d0e8082a3acf6b72d460, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:13,471 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174353471"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174353471"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174353471"}]},"ts":"1676174353471"} 2023-02-12 03:59:13,471 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174353471"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174353471"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174353471"}]},"ts":"1676174353471"} 2023-02-12 03:59:13,472 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=249, ppid=247, state=RUNNABLE; CloseRegionProcedure 5e163755ca31807434b0e014e67c9f89, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:59:13,472 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=250, ppid=244, state=RUNNABLE; CloseRegionProcedure 1991ab242a39a7de82ac275617721040, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:59:13,473 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=251, ppid=248, state=RUNNABLE; CloseRegionProcedure 35bdba3092d7a6729f7490917901687d, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:59:13,473 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=252, ppid=246, state=RUNNABLE; CloseRegionProcedure 395304cb23b0d8ab23e3ba3dc259249d, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:59:13,474 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=253, ppid=245, state=RUNNABLE; CloseRegionProcedure 8d6f7fc2ff29d0e8082a3acf6b72d460, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:59:13,475 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=ac12d10f8881ef03c212b7559a1cb27b, UNASSIGN 2023-02-12 03:59:13,475 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=ac12d10f8881ef03c212b7559a1cb27b, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:13,475 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174353475"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174353475"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174353475"}]},"ts":"1676174353475"} 2023-02-12 03:59:13,476 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=254, ppid=243, state=RUNNABLE; CloseRegionProcedure ac12d10f8881ef03c212b7559a1cb27b, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:59:13,624 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:13,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 5e163755ca31807434b0e014e67c9f89, disabling compactions & flushes 2023-02-12 03:59:13,624 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:13,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:13,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. after waiting 0 ms 2023-02-12 03:59:13,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:13,624 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 5e163755ca31807434b0e014e67c9f89 1/1 column families, dataSize=1.96 KB heapSize=4.47 KB 2023-02-12 03:59:13,625 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:13,625 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:13,625 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 395304cb23b0d8ab23e3ba3dc259249d, disabling compactions & flushes 2023-02-12 03:59:13,625 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing ac12d10f8881ef03c212b7559a1cb27b, disabling compactions & flushes 2023-02-12 03:59:13,625 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:13,625 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:13,625 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:13,626 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:13,626 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. after waiting 0 ms 2023-02-12 03:59:13,626 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. after waiting 0 ms 2023-02-12 03:59:13,626 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:13,626 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:13,626 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing ac12d10f8881ef03c212b7559a1cb27b 1/1 column families, dataSize=2.16 KB heapSize=4.89 KB 2023-02-12 03:59:13,626 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 395304cb23b0d8ab23e3ba3dc259249d 1/1 column families, dataSize=2.94 KB heapSize=6.58 KB 2023-02-12 03:59:13,649 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230212c291ab8dd9144f9583d27478af0d785a to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212c291ab8dd9144f9583d27478af0d785a 2023-02-12 03:59:13,651 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202302129ef383240168403db147f26b5bec256d to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/d41d8cd98f00b204e9800998ecf8427e202302129ef383240168403db147f26b5bec256d 2023-02-12 03:59:13,651 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302129ce77907bf8a43f6a2bfebfea85137d0 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/a87ff679a2f3e71d9181a67b7542122c202302129ce77907bf8a43f6a2bfebfea85137d0 2023-02-12 03:59:13,656 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/.tmp/cf/044a1a8620ec494e808ac6dcfe07f34f 2023-02-12 03:59:13,657 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/.tmp/cf/c203a1de813c4f6fa4f8547870b13bea 2023-02-12 03:59:13,658 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/.tmp/cf/ae19421e9ec14c2c91a5a16c0bfd026e 2023-02-12 03:59:13,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/.tmp/cf/044a1a8620ec494e808ac6dcfe07f34f as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/cf/044a1a8620ec494e808ac6dcfe07f34f 2023-02-12 03:59:13,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/.tmp/cf/ae19421e9ec14c2c91a5a16c0bfd026e as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/cf/ae19421e9ec14c2c91a5a16c0bfd026e 2023-02-12 03:59:13,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/.tmp/cf/c203a1de813c4f6fa4f8547870b13bea as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/cf/c203a1de813c4f6fa4f8547870b13bea 2023-02-12 03:59:13,662 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/cf/044a1a8620ec494e808ac6dcfe07f34f, entries=45, sequenceid=8, filesize=12.6 K 2023-02-12 03:59:13,663 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.94 KB/3013, heapSize ~6.56 KB/6720, currentSize=0 B/0 for 395304cb23b0d8ab23e3ba3dc259249d in 37ms, sequenceid=8, compaction requested=false 2023-02-12 03:59:13,663 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCorruptedSnapshot-1676174351364' 2023-02-12 03:59:13,663 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/cf/ae19421e9ec14c2c91a5a16c0bfd026e, entries=30, sequenceid=8, filesize=10.1 K 2023-02-12 03:59:13,663 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/cf/c203a1de813c4f6fa4f8547870b13bea, entries=33, sequenceid=8, filesize=10.6 K 2023-02-12 03:59:13,664 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for 5e163755ca31807434b0e014e67c9f89 in 40ms, sequenceid=8, compaction requested=false 2023-02-12 03:59:13,665 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for ac12d10f8881ef03c212b7559a1cb27b in 39ms, sequenceid=8, compaction requested=false 2023-02-12 03:59:13,667 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 03:59:13,667 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 03:59:13,667 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 03:59:13,668 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:13,668 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:13,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 395304cb23b0d8ab23e3ba3dc259249d: Waiting for close lock at 1676174353625Running coprocessor pre-close hooks at 1676174353625Disabling compacts and flushes for region at 1676174353625Disabling writes for close at 1676174353626 (+1 ms)Obtaining lock to block concurrent updates at 1676174353626Preparing flush snapshotting stores in 395304cb23b0d8ab23e3ba3dc259249d at 1676174353626Finished memstore snapshotting testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d., syncing WAL and waiting on mvcc, flushsize=dataSize=3013, getHeapSize=6720, getOffHeapSize=0, getCellsCount=45 at 1676174353627 (+1 ms)Flushing stores of testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. at 1676174353627Flushing 395304cb23b0d8ab23e3ba3dc259249d/cf: creating writer at 1676174353628 (+1 ms)Flushing mob file 395304cb23b0d8ab23e3ba3dc259249d/cf: appending metadata at 1676174353631 (+3 ms)Flushing mob file 395304cb23b0d8ab23e3ba3dc259249d/cf: closing flushed file at 1676174353631Flushing 395304cb23b0d8ab23e3ba3dc259249d/cf: appending metadata at 1676174353650 (+19 ms)Flushing 395304cb23b0d8ab23e3ba3dc259249d/cf: closing flushed file at 1676174353650Flushing 395304cb23b0d8ab23e3ba3dc259249d/cf: reopening flushed file at 1676174353660 (+10 ms)Finished flush of dataSize ~2.94 KB/3013, heapSize ~6.56 KB/6720, currentSize=0 B/0 for 395304cb23b0d8ab23e3ba3dc259249d in 37ms, sequenceid=8, compaction requested=false at 1676174353663 (+3 ms)Writing region close event to WAL at 1676174353665 (+2 ms)Running coprocessor post-close hooks at 1676174353667 (+2 ms)Closed at 1676174353668 (+1 ms) 2023-02-12 03:59:13,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 5e163755ca31807434b0e014e67c9f89: Waiting for close lock at 1676174353624Running coprocessor pre-close hooks at 1676174353624Disabling compacts and flushes for region at 1676174353624Disabling writes for close at 1676174353624Obtaining lock to block concurrent updates at 1676174353624Preparing flush snapshotting stores in 5e163755ca31807434b0e014e67c9f89 at 1676174353624Finished memstore snapshotting testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89., syncing WAL and waiting on mvcc, flushsize=dataSize=2008, getHeapSize=4560, getOffHeapSize=0, getCellsCount=30 at 1676174353625 (+1 ms)Flushing stores of testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. at 1676174353625Flushing 5e163755ca31807434b0e014e67c9f89/cf: creating writer at 1676174353626 (+1 ms)Flushing mob file 5e163755ca31807434b0e014e67c9f89/cf: appending metadata at 1676174353630 (+4 ms)Flushing mob file 5e163755ca31807434b0e014e67c9f89/cf: closing flushed file at 1676174353630Flushing 5e163755ca31807434b0e014e67c9f89/cf: appending metadata at 1676174353652 (+22 ms)Flushing 5e163755ca31807434b0e014e67c9f89/cf: closing flushed file at 1676174353652Flushing 5e163755ca31807434b0e014e67c9f89/cf: reopening flushed file at 1676174353661 (+9 ms)Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for 5e163755ca31807434b0e014e67c9f89 in 40ms, sequenceid=8, compaction requested=false at 1676174353664 (+3 ms)Writing region close event to WAL at 1676174353665 (+1 ms)Running coprocessor post-close hooks at 1676174353668 (+3 ms)Closed at 1676174353668 2023-02-12 03:59:13,668 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:13,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for ac12d10f8881ef03c212b7559a1cb27b: Waiting for close lock at 1676174353625Running coprocessor pre-close hooks at 1676174353625Disabling compacts and flushes for region at 1676174353625Disabling writes for close at 1676174353626 (+1 ms)Obtaining lock to block concurrent updates at 1676174353626Preparing flush snapshotting stores in ac12d10f8881ef03c212b7559a1cb27b at 1676174353626Finished memstore snapshotting testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b., syncing WAL and waiting on mvcc, flushsize=dataSize=2209, getHeapSize=4992, getOffHeapSize=0, getCellsCount=33 at 1676174353627 (+1 ms)Flushing stores of testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. at 1676174353627Flushing ac12d10f8881ef03c212b7559a1cb27b/cf: creating writer at 1676174353628 (+1 ms)Flushing mob file ac12d10f8881ef03c212b7559a1cb27b/cf: appending metadata at 1676174353630 (+2 ms)Flushing mob file ac12d10f8881ef03c212b7559a1cb27b/cf: closing flushed file at 1676174353630Flushing ac12d10f8881ef03c212b7559a1cb27b/cf: appending metadata at 1676174353652 (+22 ms)Flushing ac12d10f8881ef03c212b7559a1cb27b/cf: closing flushed file at 1676174353652Flushing ac12d10f8881ef03c212b7559a1cb27b/cf: reopening flushed file at 1676174353661 (+9 ms)Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for ac12d10f8881ef03c212b7559a1cb27b in 39ms, sequenceid=8, compaction requested=false at 1676174353665 (+4 ms)Writing region close event to WAL at 1676174353665Running coprocessor post-close hooks at 1676174353668 (+3 ms)Closed at 1676174353668 2023-02-12 03:59:13,669 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:13,669 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 1991ab242a39a7de82ac275617721040 2023-02-12 03:59:13,669 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 1991ab242a39a7de82ac275617721040, disabling compactions & flushes 2023-02-12 03:59:13,669 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:13,669 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:13,669 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. after waiting 0 ms 2023-02-12 03:59:13,669 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:13,669 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=5e163755ca31807434b0e014e67c9f89, regionState=CLOSED 2023-02-12 03:59:13,669 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 1991ab242a39a7de82ac275617721040 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-12 03:59:13,669 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174353669"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174353669"}]},"ts":"1676174353669"} 2023-02-12 03:59:13,670 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:13,670 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:13,670 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 8d6f7fc2ff29d0e8082a3acf6b72d460, disabling compactions & flushes 2023-02-12 03:59:13,670 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:13,670 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:13,670 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. after waiting 0 ms 2023-02-12 03:59:13,670 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:13,670 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 8d6f7fc2ff29d0e8082a3acf6b72d460 1/1 column families, dataSize=2.16 KB heapSize=4.89 KB 2023-02-12 03:59:13,672 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=395304cb23b0d8ab23e3ba3dc259249d, regionState=CLOSED 2023-02-12 03:59:13,672 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174353672"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174353672"}]},"ts":"1676174353672"} 2023-02-12 03:59:13,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:13,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:13,673 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 35bdba3092d7a6729f7490917901687d, disabling compactions & flushes 2023-02-12 03:59:13,673 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:13,673 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:13,673 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. after waiting 0 ms 2023-02-12 03:59:13,673 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:13,673 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 35bdba3092d7a6729f7490917901687d 1/1 column families, dataSize=21.13 KB heapSize=45.67 KB 2023-02-12 03:59:13,676 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=ac12d10f8881ef03c212b7559a1cb27b, regionState=CLOSED 2023-02-12 03:59:13,676 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174353676"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174353676"}]},"ts":"1676174353676"} 2023-02-12 03:59:13,681 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=249, resume processing ppid=247 2023-02-12 03:59:13,681 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=249, ppid=247, state=SUCCESS; CloseRegionProcedure 5e163755ca31807434b0e014e67c9f89, server=jenkins-hbase20.apache.org,35783,1676174315451 in 200 msec 2023-02-12 03:59:13,684 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=252, resume processing ppid=246 2023-02-12 03:59:13,684 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=252, ppid=246, state=SUCCESS; CloseRegionProcedure 395304cb23b0d8ab23e3ba3dc259249d, server=jenkins-hbase20.apache.org,42205,1676174315413 in 200 msec 2023-02-12 03:59:13,685 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=247, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=5e163755ca31807434b0e014e67c9f89, UNASSIGN in 214 msec 2023-02-12 03:59:13,685 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=254, resume processing ppid=243 2023-02-12 03:59:13,685 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=246, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=395304cb23b0d8ab23e3ba3dc259249d, UNASSIGN in 217 msec 2023-02-12 03:59:13,685 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=254, ppid=243, state=SUCCESS; CloseRegionProcedure ac12d10f8881ef03c212b7559a1cb27b, server=jenkins-hbase20.apache.org,35785,1676174315481 in 204 msec 2023-02-12 03:59:13,686 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=243, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=ac12d10f8881ef03c212b7559a1cb27b, UNASSIGN in 218 msec 2023-02-12 03:59:13,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-12 03:59:13,729 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c2023021281504bda48ef46eba3abad932559abc7 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/c81e728d9d4c2f636f067f89cc14862c2023021281504bda48ef46eba3abad932559abc7 2023-02-12 03:59:13,738 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230212474eeddee91647a29b4bb8010100f7a3 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/e4da3b7fbbce2345d7772b0674a318d520230212474eeddee91647a29b4bb8010100f7a3 2023-02-12 03:59:13,743 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/.tmp/cf/107ee1abf95441bb87814c20220acd44 2023-02-12 03:59:13,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b202302122fccebde9fc74f9e8335c7d37fc7a3bc to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/c4ca4238a0b923820dcc509a6f75849b202302122fccebde9fc74f9e8335c7d37fc7a3bc 2023-02-12 03:59:13,747 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/.tmp/cf/107ee1abf95441bb87814c20220acd44 as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/cf/107ee1abf95441bb87814c20220acd44 2023-02-12 03:59:13,748 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=21.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/.tmp/cf/e58e4a3220514ebaa33fd06e0beba2b6 2023-02-12 03:59:13,750 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/cf/107ee1abf95441bb87814c20220acd44, entries=33, sequenceid=8, filesize=10.6 K 2023-02-12 03:59:13,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for 8d6f7fc2ff29d0e8082a3acf6b72d460 in 81ms, sequenceid=8, compaction requested=false 2023-02-12 03:59:13,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/.tmp/cf/002ae5f09d194b26bbdce801282de0b5 2023-02-12 03:59:13,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/.tmp/cf/e58e4a3220514ebaa33fd06e0beba2b6 as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/cf/e58e4a3220514ebaa33fd06e0beba2b6 2023-02-12 03:59:13,753 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 03:59:13,754 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:13,754 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 8d6f7fc2ff29d0e8082a3acf6b72d460: Waiting for close lock at 1676174353670Running coprocessor pre-close hooks at 1676174353670Disabling compacts and flushes for region at 1676174353670Disabling writes for close at 1676174353670Obtaining lock to block concurrent updates at 1676174353671 (+1 ms)Preparing flush snapshotting stores in 8d6f7fc2ff29d0e8082a3acf6b72d460 at 1676174353671Finished memstore snapshotting testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460., syncing WAL and waiting on mvcc, flushsize=dataSize=2209, getHeapSize=4992, getOffHeapSize=0, getCellsCount=33 at 1676174353671Flushing stores of testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. at 1676174353676 (+5 ms)Flushing 8d6f7fc2ff29d0e8082a3acf6b72d460/cf: creating writer at 1676174353676Flushing mob file 8d6f7fc2ff29d0e8082a3acf6b72d460/cf: appending metadata at 1676174353683 (+7 ms)Flushing mob file 8d6f7fc2ff29d0e8082a3acf6b72d460/cf: closing flushed file at 1676174353683Flushing 8d6f7fc2ff29d0e8082a3acf6b72d460/cf: appending metadata at 1676174353730 (+47 ms)Flushing 8d6f7fc2ff29d0e8082a3acf6b72d460/cf: closing flushed file at 1676174353730Flushing 8d6f7fc2ff29d0e8082a3acf6b72d460/cf: reopening flushed file at 1676174353748 (+18 ms)Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for 8d6f7fc2ff29d0e8082a3acf6b72d460 in 81ms, sequenceid=8, compaction requested=false at 1676174353751 (+3 ms)Writing region close event to WAL at 1676174353751Running coprocessor post-close hooks at 1676174353754 (+3 ms)Closed at 1676174353754 2023-02-12 03:59:13,754 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/.tmp/cf/002ae5f09d194b26bbdce801282de0b5 as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/cf/002ae5f09d194b26bbdce801282de0b5 2023-02-12 03:59:13,754 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/cf/e58e4a3220514ebaa33fd06e0beba2b6, entries=323, sequenceid=8, filesize=60.6 K 2023-02-12 03:59:13,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~21.13 KB/21637, heapSize ~45.66 KB/46752, currentSize=0 B/0 for 35bdba3092d7a6729f7490917901687d in 82ms, sequenceid=8, compaction requested=false 2023-02-12 03:59:13,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:13,755 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=8d6f7fc2ff29d0e8082a3acf6b72d460, regionState=CLOSED 2023-02-12 03:59:13,755 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174353755"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174353755"}]},"ts":"1676174353755"} 2023-02-12 03:59:13,757 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/cf/002ae5f09d194b26bbdce801282de0b5, entries=36, sequenceid=8, filesize=11.1 K 2023-02-12 03:59:13,758 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 1991ab242a39a7de82ac275617721040 in 89ms, sequenceid=8, compaction requested=false 2023-02-12 03:59:13,758 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 03:59:13,758 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:13,758 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 35bdba3092d7a6729f7490917901687d: Waiting for close lock at 1676174353673Running coprocessor pre-close hooks at 1676174353673Disabling compacts and flushes for region at 1676174353673Disabling writes for close at 1676174353673Obtaining lock to block concurrent updates at 1676174353673Preparing flush snapshotting stores in 35bdba3092d7a6729f7490917901687d at 1676174353673Finished memstore snapshotting testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d., syncing WAL and waiting on mvcc, flushsize=dataSize=21637, getHeapSize=46752, getOffHeapSize=0, getCellsCount=323 at 1676174353673Flushing stores of testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. at 1676174353673Flushing 35bdba3092d7a6729f7490917901687d/cf: creating writer at 1676174353676 (+3 ms)Flushing mob file 35bdba3092d7a6729f7490917901687d/cf: appending metadata at 1676174353683 (+7 ms)Flushing mob file 35bdba3092d7a6729f7490917901687d/cf: closing flushed file at 1676174353683Flushing 35bdba3092d7a6729f7490917901687d/cf: appending metadata at 1676174353739 (+56 ms)Flushing 35bdba3092d7a6729f7490917901687d/cf: closing flushed file at 1676174353739Flushing 35bdba3092d7a6729f7490917901687d/cf: reopening flushed file at 1676174353752 (+13 ms)Finished flush of dataSize ~21.13 KB/21637, heapSize ~45.66 KB/46752, currentSize=0 B/0 for 35bdba3092d7a6729f7490917901687d in 82ms, sequenceid=8, compaction requested=false at 1676174353755 (+3 ms)Writing region close event to WAL at 1676174353755Running coprocessor post-close hooks at 1676174353758 (+3 ms)Closed at 1676174353758 2023-02-12 03:59:13,759 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:13,760 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=35bdba3092d7a6729f7490917901687d, regionState=CLOSED 2023-02-12 03:59:13,760 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=253, resume processing ppid=245 2023-02-12 03:59:13,760 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174353760"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174353760"}]},"ts":"1676174353760"} 2023-02-12 03:59:13,760 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=253, ppid=245, state=SUCCESS; CloseRegionProcedure 8d6f7fc2ff29d0e8082a3acf6b72d460, server=jenkins-hbase20.apache.org,42205,1676174315413 in 283 msec 2023-02-12 03:59:13,762 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=245, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=8d6f7fc2ff29d0e8082a3acf6b72d460, UNASSIGN in 293 msec 2023-02-12 03:59:13,763 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 03:59:13,763 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=251, resume processing ppid=248 2023-02-12 03:59:13,763 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=251, ppid=248, state=SUCCESS; CloseRegionProcedure 35bdba3092d7a6729f7490917901687d, server=jenkins-hbase20.apache.org,35785,1676174315481 in 288 msec 2023-02-12 03:59:13,763 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:13,763 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 1991ab242a39a7de82ac275617721040: Waiting for close lock at 1676174353669Running coprocessor pre-close hooks at 1676174353669Disabling compacts and flushes for region at 1676174353669Disabling writes for close at 1676174353669Obtaining lock to block concurrent updates at 1676174353669Preparing flush snapshotting stores in 1991ab242a39a7de82ac275617721040 at 1676174353669Finished memstore snapshotting testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040., syncing WAL and waiting on mvcc, flushsize=dataSize=2410, getHeapSize=5424, getOffHeapSize=0, getCellsCount=36 at 1676174353669Flushing stores of testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. at 1676174353670 (+1 ms)Flushing 1991ab242a39a7de82ac275617721040/cf: creating writer at 1676174353671 (+1 ms)Flushing mob file 1991ab242a39a7de82ac275617721040/cf: appending metadata at 1676174353680 (+9 ms)Flushing mob file 1991ab242a39a7de82ac275617721040/cf: closing flushed file at 1676174353680Flushing 1991ab242a39a7de82ac275617721040/cf: appending metadata at 1676174353745 (+65 ms)Flushing 1991ab242a39a7de82ac275617721040/cf: closing flushed file at 1676174353745Flushing 1991ab242a39a7de82ac275617721040/cf: reopening flushed file at 1676174353754 (+9 ms)Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 1991ab242a39a7de82ac275617721040 in 89ms, sequenceid=8, compaction requested=false at 1676174353758 (+4 ms)Writing region close event to WAL at 1676174353758Running coprocessor post-close hooks at 1676174353763 (+5 ms)Closed at 1676174353763 2023-02-12 03:59:13,764 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=248, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=35bdba3092d7a6729f7490917901687d, UNASSIGN in 296 msec 2023-02-12 03:59:13,764 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 1991ab242a39a7de82ac275617721040 2023-02-12 03:59:13,764 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=1991ab242a39a7de82ac275617721040, regionState=CLOSED 2023-02-12 03:59:13,765 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174353764"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174353764"}]},"ts":"1676174353764"} 2023-02-12 03:59:13,766 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=250, resume processing ppid=244 2023-02-12 03:59:13,766 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=250, ppid=244, state=SUCCESS; CloseRegionProcedure 1991ab242a39a7de82ac275617721040, server=jenkins-hbase20.apache.org,35783,1676174315451 in 293 msec 2023-02-12 03:59:13,767 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=244, resume processing ppid=242 2023-02-12 03:59:13,767 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=244, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=1991ab242a39a7de82ac275617721040, UNASSIGN in 299 msec 2023-02-12 03:59:13,768 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676174351364","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174353768"}]},"ts":"1676174353768"} 2023-02-12 03:59:13,769 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676174351364, state=DISABLED in hbase:meta 2023-02-12 03:59:13,770 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676174351364 to state=DISABLED 2023-02-12 03:59:13,771 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=242, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676174351364 in 308 msec 2023-02-12 03:59:14,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-12 03:59:14,217 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676174351364, procId: 242 completed 2023-02-12 03:59:14,218 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1580): Client=jenkins//148.251.75.209 snapshot request for:{ ss=snaptb0-1676174351364 table=testCorruptedSnapshot-1676174351364 type=FLUSH ttl=0 } 2023-02-12 03:59:14,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676174354218 (current time:1676174354218). 2023-02-12 03:59:14,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-12 03:59:14,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] zookeeper.ReadOnlyZKClient(139): Connect 0x19f412eb to 127.0.0.1:50368 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 03:59:14,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@402323a2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 03:59:14,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 03:59:14,227 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42908, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 03:59:14,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x19f412eb to 127.0.0.1:50368 2023-02-12 03:59:14,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:59:14,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-12 03:59:14,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676174351364 table=testCorruptedSnapshot-1676174351364 type=FLUSH ttl=0 } 2023-02-12 03:59:14,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676174351364, type=EXCLUSIVE 2023-02-12 03:59:14,231 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676174351364, type=EXCLUSIVE 2023-02-12 03:59:14,232 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=255, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676174351364, type=EXCLUSIVE; timeout=600000, timestamp=1676174954232 2023-02-12 03:59:14,232 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676174351364 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676174351364 2023-02-12 03:59:14,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676174351364 table=testCorruptedSnapshot-1676174351364 type=FLUSH ttl=0 } 2023-02-12 03:59:14,232 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676174351364 table=testCorruptedSnapshot-1676174351364 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-12 03:59:14,232 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-12 03:59:14,244 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676174351364 table=testCorruptedSnapshot-1676174351364 type=DISABLED ttl=0 } 2023-02-12 03:59:14,246 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:59:14,246 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:59:14,246 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:59:14,246 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:59:14,246 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:59:14,246 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:59:14,246 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:59:14,246 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:59:14,246 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:59:14,246 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 03:59:14,246 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:59:14,246 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:59:14,246 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:59:14,246 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 03:59:14,248 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/cf/e58e4a3220514ebaa33fd06e0beba2b6] hfiles 2023-02-12 03:59:14,248 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/cf/e58e4a3220514ebaa33fd06e0beba2b6 2023-02-12 03:59:14,248 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/cf/107ee1abf95441bb87814c20220acd44] hfiles 2023-02-12 03:59:14,248 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/cf/002ae5f09d194b26bbdce801282de0b5] hfiles 2023-02-12 03:59:14,248 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/cf/107ee1abf95441bb87814c20220acd44 2023-02-12 03:59:14,248 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/cf/002ae5f09d194b26bbdce801282de0b5 2023-02-12 03:59:14,248 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/cf/044a1a8620ec494e808ac6dcfe07f34f] hfiles 2023-02-12 03:59:14,248 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/cf/c203a1de813c4f6fa4f8547870b13bea] hfiles 2023-02-12 03:59:14,248 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/cf/044a1a8620ec494e808ac6dcfe07f34f 2023-02-12 03:59:14,248 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/cf/c203a1de813c4f6fa4f8547870b13bea 2023-02-12 03:59:14,253 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/cf/ae19421e9ec14c2c91a5a16c0bfd026e] hfiles 2023-02-12 03:59:14,253 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/cf/ae19421e9ec14c2c91a5a16c0bfd026e 2023-02-12 03:59:14,255 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/a87ff679a2f3e71d9181a67b7542122c202302129ce77907bf8a43f6a2bfebfea85137d0, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/c4ca4238a0b923820dcc509a6f75849b202302122fccebde9fc74f9e8335c7d37fc7a3bc, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/c81e728d9d4c2f636f067f89cc14862c2023021281504bda48ef46eba3abad932559abc7, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/d41d8cd98f00b204e9800998ecf8427e202302129ef383240168403db147f26b5bec256d, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/e4da3b7fbbce2345d7772b0674a318d520230212474eeddee91647a29b4bb8010100f7a3, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212c291ab8dd9144f9583d27478af0d785a] hfiles 2023-02-12 03:59:14,256 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/a87ff679a2f3e71d9181a67b7542122c202302129ce77907bf8a43f6a2bfebfea85137d0 2023-02-12 03:59:14,256 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/c4ca4238a0b923820dcc509a6f75849b202302122fccebde9fc74f9e8335c7d37fc7a3bc 2023-02-12 03:59:14,256 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/c81e728d9d4c2f636f067f89cc14862c2023021281504bda48ef46eba3abad932559abc7 2023-02-12 03:59:14,256 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/d41d8cd98f00b204e9800998ecf8427e202302129ef383240168403db147f26b5bec256d 2023-02-12 03:59:14,256 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/e4da3b7fbbce2345d7772b0674a318d520230212474eeddee91647a29b4bb8010100f7a3 2023-02-12 03:59:14,256 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212c291ab8dd9144f9583d27478af0d785a 2023-02-12 03:59:14,482 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 03:59:14,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676174351364 table=testCorruptedSnapshot-1676174351364 type=FLUSH ttl=0 } is done 2023-02-12 03:59:14,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(404): Snapshoting '{ ss=snaptb0-1676174351364 table=testCorruptedSnapshot-1676174351364 type=FLUSH ttl=0 }' is still in progress! 2023-02-12 03:59:14,484 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-12 03:59:14,670 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676174351364 table=testCorruptedSnapshot-1676174351364 type=DISABLED ttl=0 } as finished. 2023-02-12 03:59:14,670 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676174351364 2023-02-12 03:59:14,670 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/snaptb0-1676174351364 2023-02-12 03:59:14,691 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/snaptb0-1676174351364 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/snaptb0-1676174351364 2023-02-12 03:59:14,707 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676174351364 of table testCorruptedSnapshot-1676174351364 completed 2023-02-12 03:59:14,707 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/snaptb0-1676174351364 2023-02-12 03:59:14,708 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.hbase-snapshot/.tmp/snaptb0-1676174351364 2023-02-12 03:59:14,708 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676174351364 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676174351364 at 1676174354232Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676174351364 table=testCorruptedSnapshot-1676174351364 type=DISABLED ttl=0 } at 1676174354244 (+12 ms)Consolidate snapshot: snaptb0-1676174351364 at 1676174354670 (+426 ms)Loading Region manifests for snaptb0-1676174351364 at 1676174354670Writing data manifest for snaptb0-1676174351364 at 1676174354674 (+4 ms)Verifying snapshot: snaptb0-1676174351364 at 1676174354686 (+12 ms)Snapshot snaptb0-1676174351364 of table testCorruptedSnapshot-1676174351364 completed at 1676174354707 (+21 ms) 2023-02-12 03:59:14,709 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=255, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676174351364, type=EXCLUSIVE 2023-02-12 03:59:14,710 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=255, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676174351364, type=EXCLUSIVE in 479 msec 2023-02-12 03:59:14,984 DEBUG [Listener at localhost.localdomain/46483] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 03:59:14,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676174351364 table=testCorruptedSnapshot-1676174351364 type=FLUSH ttl=0 } is done 2023-02-12 03:59:14,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676174351364 table=testCorruptedSnapshot-1676174351364 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-12 03:59:14,987 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676174351364 2023-02-12 03:59:14,987 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$10(2372): Client=jenkins//148.251.75.209 enable testCorruptedSnapshot-1676174351364 2023-02-12 03:59:14,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=256, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676174351364 2023-02-12 03:59:14,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-12 03:59:14,993 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676174351364 2023-02-12 03:59:14,994 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676174351364","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174354994"}]},"ts":"1676174354994"} 2023-02-12 03:59:14,995 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676174351364, state=ENABLING in hbase:meta 2023-02-12 03:59:14,997 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-12 03:59:14,997 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=ac12d10f8881ef03c212b7559a1cb27b, ASSIGN}, {pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=1991ab242a39a7de82ac275617721040, ASSIGN}, {pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=8d6f7fc2ff29d0e8082a3acf6b72d460, ASSIGN}, {pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=395304cb23b0d8ab23e3ba3dc259249d, ASSIGN}, {pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=5e163755ca31807434b0e014e67c9f89, ASSIGN}, {pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=35bdba3092d7a6729f7490917901687d, ASSIGN}] 2023-02-12 03:59:14,998 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=35bdba3092d7a6729f7490917901687d, ASSIGN 2023-02-12 03:59:14,998 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=395304cb23b0d8ab23e3ba3dc259249d, ASSIGN 2023-02-12 03:59:14,998 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=5e163755ca31807434b0e014e67c9f89, ASSIGN 2023-02-12 03:59:14,998 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=8d6f7fc2ff29d0e8082a3acf6b72d460, ASSIGN 2023-02-12 03:59:14,999 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=1991ab242a39a7de82ac275617721040, ASSIGN 2023-02-12 03:59:14,999 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:14,999 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:14,999 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:14,999 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=35bdba3092d7a6729f7490917901687d, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,35785,1676174315481; forceNewPlan=false, retain=true 2023-02-12 03:59:14,999 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=395304cb23b0d8ab23e3ba3dc259249d, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,42205,1676174315413; forceNewPlan=false, retain=true 2023-02-12 03:59:14,999 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:14,999 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=5e163755ca31807434b0e014e67c9f89, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,35783,1676174315451; forceNewPlan=false, retain=true 2023-02-12 03:59:14,999 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:14,999 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=8d6f7fc2ff29d0e8082a3acf6b72d460, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,42205,1676174315413; forceNewPlan=false, retain=true 2023-02-12 03:59:14,999 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=1991ab242a39a7de82ac275617721040, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,35783,1676174315451; forceNewPlan=false, retain=true 2023-02-12 03:59:15,000 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=ac12d10f8881ef03c212b7559a1cb27b, ASSIGN 2023-02-12 03:59:15,000 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:15,000 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=ac12d10f8881ef03c212b7559a1cb27b, ASSIGN; state=CLOSED, location=jenkins-hbase20.apache.org,35785,1676174315481; forceNewPlan=false, retain=true 2023-02-12 03:59:15,149 INFO [jenkins-hbase20:37305] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-12 03:59:15,150 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=8d6f7fc2ff29d0e8082a3acf6b72d460, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:15,150 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=5e163755ca31807434b0e014e67c9f89, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:15,150 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174355150"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174355150"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174355150"}]},"ts":"1676174355150"} 2023-02-12 03:59:15,151 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174355150"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174355150"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174355150"}]},"ts":"1676174355150"} 2023-02-12 03:59:15,150 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=35bdba3092d7a6729f7490917901687d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:15,150 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=ac12d10f8881ef03c212b7559a1cb27b, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:15,150 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=395304cb23b0d8ab23e3ba3dc259249d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:15,151 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174355150"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174355150"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174355150"}]},"ts":"1676174355150"} 2023-02-12 03:59:15,151 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174355150"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174355150"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174355150"}]},"ts":"1676174355150"} 2023-02-12 03:59:15,152 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174355150"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174355150"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174355150"}]},"ts":"1676174355150"} 2023-02-12 03:59:15,154 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=263, ppid=259, state=RUNNABLE; OpenRegionProcedure 8d6f7fc2ff29d0e8082a3acf6b72d460, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:59:15,155 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=264, ppid=261, state=RUNNABLE; OpenRegionProcedure 5e163755ca31807434b0e014e67c9f89, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:59:15,156 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=265, ppid=257, state=RUNNABLE; OpenRegionProcedure ac12d10f8881ef03c212b7559a1cb27b, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:59:15,158 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=266, ppid=262, state=RUNNABLE; OpenRegionProcedure 35bdba3092d7a6729f7490917901687d, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:59:15,158 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=267, ppid=260, state=RUNNABLE; OpenRegionProcedure 395304cb23b0d8ab23e3ba3dc259249d, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:59:15,159 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=1991ab242a39a7de82ac275617721040, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:15,159 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174355158"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174355158"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174355158"}]},"ts":"1676174355158"} 2023-02-12 03:59:15,160 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=268, ppid=258, state=RUNNABLE; OpenRegionProcedure 1991ab242a39a7de82ac275617721040, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:59:15,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-12 03:59:15,309 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:15,309 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:15,309 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 8d6f7fc2ff29d0e8082a3acf6b72d460, NAME => 'testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.', STARTKEY => '2', ENDKEY => '3'} 2023-02-12 03:59:15,309 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5e163755ca31807434b0e014e67c9f89, NAME => 'testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.', STARTKEY => '4', ENDKEY => '5'} 2023-02-12 03:59:15,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676174351364 8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:15,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676174351364 5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:15,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:15,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:15,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:15,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:15,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:15,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:15,311 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:15,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 35bdba3092d7a6729f7490917901687d, NAME => 'testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.', STARTKEY => '5', ENDKEY => ''} 2023-02-12 03:59:15,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676174351364 35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:15,311 INFO [StoreOpener-5e163755ca31807434b0e014e67c9f89-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:15,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:15,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:15,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:15,312 INFO [StoreOpener-8d6f7fc2ff29d0e8082a3acf6b72d460-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:15,312 DEBUG [StoreOpener-5e163755ca31807434b0e014e67c9f89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/cf 2023-02-12 03:59:15,312 DEBUG [StoreOpener-5e163755ca31807434b0e014e67c9f89-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/cf 2023-02-12 03:59:15,312 INFO [StoreOpener-35bdba3092d7a6729f7490917901687d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:15,313 INFO [StoreOpener-5e163755ca31807434b0e014e67c9f89-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5e163755ca31807434b0e014e67c9f89 columnFamilyName cf 2023-02-12 03:59:15,313 DEBUG [StoreOpener-8d6f7fc2ff29d0e8082a3acf6b72d460-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/cf 2023-02-12 03:59:15,313 DEBUG [StoreOpener-8d6f7fc2ff29d0e8082a3acf6b72d460-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/cf 2023-02-12 03:59:15,313 DEBUG [StoreOpener-35bdba3092d7a6729f7490917901687d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/cf 2023-02-12 03:59:15,313 DEBUG [StoreOpener-35bdba3092d7a6729f7490917901687d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/cf 2023-02-12 03:59:15,313 INFO [StoreOpener-8d6f7fc2ff29d0e8082a3acf6b72d460-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8d6f7fc2ff29d0e8082a3acf6b72d460 columnFamilyName cf 2023-02-12 03:59:15,313 INFO [StoreOpener-35bdba3092d7a6729f7490917901687d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 35bdba3092d7a6729f7490917901687d columnFamilyName cf 2023-02-12 03:59:15,320 DEBUG [StoreOpener-5e163755ca31807434b0e014e67c9f89-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/cf/ae19421e9ec14c2c91a5a16c0bfd026e 2023-02-12 03:59:15,321 INFO [StoreOpener-5e163755ca31807434b0e014e67c9f89-1] regionserver.HStore(310): Store=5e163755ca31807434b0e014e67c9f89/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:15,321 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:15,322 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:15,322 DEBUG [StoreOpener-35bdba3092d7a6729f7490917901687d-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/cf/e58e4a3220514ebaa33fd06e0beba2b6 2023-02-12 03:59:15,322 INFO [StoreOpener-35bdba3092d7a6729f7490917901687d-1] regionserver.HStore(310): Store=35bdba3092d7a6729f7490917901687d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:15,323 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:15,323 DEBUG [StoreOpener-8d6f7fc2ff29d0e8082a3acf6b72d460-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/cf/107ee1abf95441bb87814c20220acd44 2023-02-12 03:59:15,323 INFO [StoreOpener-8d6f7fc2ff29d0e8082a3acf6b72d460-1] regionserver.HStore(310): Store=8d6f7fc2ff29d0e8082a3acf6b72d460/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:15,324 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:15,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:15,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:15,326 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 5e163755ca31807434b0e014e67c9f89; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68051943, jitterRate=0.014052972197532654}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:15,326 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 5e163755ca31807434b0e014e67c9f89: Running coprocessor pre-open hook at 1676174355310Writing region info on filesystem at 1676174355310Initializing all the Stores at 1676174355311 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174355311Cleaning up temporary data from old regions at 1676174355322 (+11 ms)Cleaning up detritus from prior splits at 1676174355323 (+1 ms)Running coprocessor post-open hooks at 1676174355326 (+3 ms)Region opened successfully at 1676174355326 2023-02-12 03:59:15,326 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:15,328 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:15,328 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89., pid=264, masterSystemTime=1676174355307 2023-02-12 03:59:15,329 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 35bdba3092d7a6729f7490917901687d; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60413860, jitterRate=-0.09976333379745483}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:15,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 35bdba3092d7a6729f7490917901687d: Running coprocessor pre-open hook at 1676174355312Writing region info on filesystem at 1676174355312Initializing all the Stores at 1676174355312Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174355312Cleaning up temporary data from old regions at 1676174355325 (+13 ms)Cleaning up detritus from prior splits at 1676174355326 (+1 ms)Running coprocessor post-open hooks at 1676174355330 (+4 ms)Region opened successfully at 1676174355330 2023-02-12 03:59:15,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:15,330 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:15,330 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:15,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:15,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1991ab242a39a7de82ac275617721040, NAME => 'testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.', STARTKEY => '1', ENDKEY => '2'} 2023-02-12 03:59:15,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676174351364 1991ab242a39a7de82ac275617721040 2023-02-12 03:59:15,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:15,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 1991ab242a39a7de82ac275617721040 2023-02-12 03:59:15,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 1991ab242a39a7de82ac275617721040 2023-02-12 03:59:15,330 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=5e163755ca31807434b0e014e67c9f89, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:15,331 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d., pid=266, masterSystemTime=1676174355308 2023-02-12 03:59:15,331 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174355330"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174355330"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174355330"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174355330"}]},"ts":"1676174355330"} 2023-02-12 03:59:15,331 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 8d6f7fc2ff29d0e8082a3acf6b72d460; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67986725, jitterRate=0.013081148266792297}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:15,331 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 8d6f7fc2ff29d0e8082a3acf6b72d460: Running coprocessor pre-open hook at 1676174355311Writing region info on filesystem at 1676174355311Initializing all the Stores at 1676174355312 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174355312Cleaning up temporary data from old regions at 1676174355327 (+15 ms)Cleaning up detritus from prior splits at 1676174355328 (+1 ms)Running coprocessor post-open hooks at 1676174355331 (+3 ms)Region opened successfully at 1676174355331 2023-02-12 03:59:15,331 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460., pid=263, masterSystemTime=1676174355307 2023-02-12 03:59:15,332 INFO [StoreOpener-1991ab242a39a7de82ac275617721040-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1991ab242a39a7de82ac275617721040 2023-02-12 03:59:15,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:15,332 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:15,333 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:15,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ac12d10f8881ef03c212b7559a1cb27b, NAME => 'testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.', STARTKEY => '', ENDKEY => '1'} 2023-02-12 03:59:15,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676174351364 ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:15,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:15,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:15,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:15,333 DEBUG [StoreOpener-1991ab242a39a7de82ac275617721040-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/cf 2023-02-12 03:59:15,333 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=35bdba3092d7a6729f7490917901687d, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:15,334 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174355333"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174355333"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174355333"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174355333"}]},"ts":"1676174355333"} 2023-02-12 03:59:15,334 DEBUG [StoreOpener-1991ab242a39a7de82ac275617721040-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/cf 2023-02-12 03:59:15,334 INFO [StoreOpener-1991ab242a39a7de82ac275617721040-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1991ab242a39a7de82ac275617721040 columnFamilyName cf 2023-02-12 03:59:15,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:15,336 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:15,336 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:15,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 395304cb23b0d8ab23e3ba3dc259249d, NAME => 'testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.', STARTKEY => '3', ENDKEY => '4'} 2023-02-12 03:59:15,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676174351364 395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:15,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 03:59:15,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7890): checking encryption for 395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:15,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7893): checking classloading for 395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:15,337 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=8d6f7fc2ff29d0e8082a3acf6b72d460, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:15,337 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174355337"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174355337"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174355337"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174355337"}]},"ts":"1676174355337"} 2023-02-12 03:59:15,337 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=264, resume processing ppid=261 2023-02-12 03:59:15,337 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=264, ppid=261, state=SUCCESS; OpenRegionProcedure 5e163755ca31807434b0e014e67c9f89, server=jenkins-hbase20.apache.org,35783,1676174315451 in 178 msec 2023-02-12 03:59:15,339 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=261, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=5e163755ca31807434b0e014e67c9f89, ASSIGN in 340 msec 2023-02-12 03:59:15,340 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=266, resume processing ppid=262 2023-02-12 03:59:15,340 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=266, ppid=262, state=SUCCESS; OpenRegionProcedure 35bdba3092d7a6729f7490917901687d, server=jenkins-hbase20.apache.org,35785,1676174315481 in 179 msec 2023-02-12 03:59:15,341 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=263, resume processing ppid=259 2023-02-12 03:59:15,341 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=262, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=35bdba3092d7a6729f7490917901687d, ASSIGN in 343 msec 2023-02-12 03:59:15,341 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=263, ppid=259, state=SUCCESS; OpenRegionProcedure 8d6f7fc2ff29d0e8082a3acf6b72d460, server=jenkins-hbase20.apache.org,42205,1676174315413 in 185 msec 2023-02-12 03:59:15,342 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=259, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=8d6f7fc2ff29d0e8082a3acf6b72d460, ASSIGN in 344 msec 2023-02-12 03:59:15,346 INFO [StoreOpener-ac12d10f8881ef03c212b7559a1cb27b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:15,347 INFO [StoreOpener-395304cb23b0d8ab23e3ba3dc259249d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:15,349 DEBUG [StoreOpener-1991ab242a39a7de82ac275617721040-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/cf/002ae5f09d194b26bbdce801282de0b5 2023-02-12 03:59:15,349 INFO [StoreOpener-1991ab242a39a7de82ac275617721040-1] regionserver.HStore(310): Store=1991ab242a39a7de82ac275617721040/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:15,351 DEBUG [StoreOpener-ac12d10f8881ef03c212b7559a1cb27b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/cf 2023-02-12 03:59:15,351 DEBUG [StoreOpener-395304cb23b0d8ab23e3ba3dc259249d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/cf 2023-02-12 03:59:15,351 DEBUG [StoreOpener-ac12d10f8881ef03c212b7559a1cb27b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/cf 2023-02-12 03:59:15,351 DEBUG [StoreOpener-395304cb23b0d8ab23e3ba3dc259249d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/cf 2023-02-12 03:59:15,352 INFO [StoreOpener-ac12d10f8881ef03c212b7559a1cb27b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ac12d10f8881ef03c212b7559a1cb27b columnFamilyName cf 2023-02-12 03:59:15,352 INFO [StoreOpener-395304cb23b0d8ab23e3ba3dc259249d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 395304cb23b0d8ab23e3ba3dc259249d columnFamilyName cf 2023-02-12 03:59:15,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040 2023-02-12 03:59:15,355 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040 2023-02-12 03:59:15,358 DEBUG [StoreOpener-395304cb23b0d8ab23e3ba3dc259249d-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/cf/044a1a8620ec494e808ac6dcfe07f34f 2023-02-12 03:59:15,358 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 1991ab242a39a7de82ac275617721040 2023-02-12 03:59:15,358 INFO [StoreOpener-395304cb23b0d8ab23e3ba3dc259249d-1] regionserver.HStore(310): Store=395304cb23b0d8ab23e3ba3dc259249d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:15,358 DEBUG [StoreOpener-ac12d10f8881ef03c212b7559a1cb27b-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/cf/c203a1de813c4f6fa4f8547870b13bea 2023-02-12 03:59:15,358 INFO [StoreOpener-ac12d10f8881ef03c212b7559a1cb27b-1] regionserver.HStore(310): Store=ac12d10f8881ef03c212b7559a1cb27b/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 03:59:15,359 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 1991ab242a39a7de82ac275617721040; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73626164, jitterRate=0.09711533784866333}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:15,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 1991ab242a39a7de82ac275617721040: Running coprocessor pre-open hook at 1676174355330Writing region info on filesystem at 1676174355330Initializing all the Stores at 1676174355331 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174355331Cleaning up temporary data from old regions at 1676174355356 (+25 ms)Cleaning up detritus from prior splits at 1676174355356Running coprocessor post-open hooks at 1676174355359 (+3 ms)Region opened successfully at 1676174355359 2023-02-12 03:59:15,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:15,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:15,359 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040., pid=268, masterSystemTime=1676174355307 2023-02-12 03:59:15,360 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:15,360 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:15,360 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:15,360 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:15,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for 395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:15,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1054): writing seq id for ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:15,362 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened 395304cb23b0d8ab23e3ba3dc259249d; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65590188, jitterRate=-0.022630035877227783}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:15,362 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1071): Opened ac12d10f8881ef03c212b7559a1cb27b; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61173603, jitterRate=-0.08844228088855743}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 03:59:15,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for 395304cb23b0d8ab23e3ba3dc259249d: Running coprocessor pre-open hook at 1676174355336Writing region info on filesystem at 1676174355336Initializing all the Stores at 1676174355338 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174355338Cleaning up temporary data from old regions at 1676174355360 (+22 ms)Cleaning up detritus from prior splits at 1676174355360Running coprocessor post-open hooks at 1676174355362 (+2 ms)Region opened successfully at 1676174355362 2023-02-12 03:59:15,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(964): Region open journal for ac12d10f8881ef03c212b7559a1cb27b: Running coprocessor pre-open hook at 1676174355333Writing region info on filesystem at 1676174355333Initializing all the Stores at 1676174355336 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676174355336Cleaning up temporary data from old regions at 1676174355360 (+24 ms)Cleaning up detritus from prior splits at 1676174355360Running coprocessor post-open hooks at 1676174355362 (+2 ms)Region opened successfully at 1676174355362 2023-02-12 03:59:15,363 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d., pid=267, masterSystemTime=1676174355307 2023-02-12 03:59:15,363 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b., pid=265, masterSystemTime=1676174355308 2023-02-12 03:59:15,365 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=1991ab242a39a7de82ac275617721040, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:15,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:15,365 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174355365"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174355365"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174355365"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174355365"}]},"ts":"1676174355365"} 2023-02-12 03:59:15,365 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:15,366 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=ac12d10f8881ef03c212b7559a1cb27b, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:15,366 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174355366"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174355366"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174355366"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174355366"}]},"ts":"1676174355366"} 2023-02-12 03:59:15,366 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:15,366 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:15,366 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=395304cb23b0d8ab23e3ba3dc259249d, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:15,366 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174355366"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1676174355366"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676174355366"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676174355366"}]},"ts":"1676174355366"} 2023-02-12 03:59:15,370 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=265, resume processing ppid=257 2023-02-12 03:59:15,370 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=268, resume processing ppid=258 2023-02-12 03:59:15,370 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=265, ppid=257, state=SUCCESS; OpenRegionProcedure ac12d10f8881ef03c212b7559a1cb27b, server=jenkins-hbase20.apache.org,35785,1676174315481 in 211 msec 2023-02-12 03:59:15,370 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=268, ppid=258, state=SUCCESS; OpenRegionProcedure 1991ab242a39a7de82ac275617721040, server=jenkins-hbase20.apache.org,35783,1676174315451 in 206 msec 2023-02-12 03:59:15,370 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=267, resume processing ppid=260 2023-02-12 03:59:15,370 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=267, ppid=260, state=SUCCESS; OpenRegionProcedure 395304cb23b0d8ab23e3ba3dc259249d, server=jenkins-hbase20.apache.org,42205,1676174315413 in 209 msec 2023-02-12 03:59:15,371 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=260, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=395304cb23b0d8ab23e3ba3dc259249d, ASSIGN in 373 msec 2023-02-12 03:59:15,371 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=258, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=1991ab242a39a7de82ac275617721040, ASSIGN in 373 msec 2023-02-12 03:59:15,371 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=257, resume processing ppid=256 2023-02-12 03:59:15,371 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=257, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=ac12d10f8881ef03c212b7559a1cb27b, ASSIGN in 373 msec 2023-02-12 03:59:15,372 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676174351364","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174355372"}]},"ts":"1676174355372"} 2023-02-12 03:59:15,373 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676174351364, state=ENABLED in hbase:meta 2023-02-12 03:59:15,374 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676174351364' was successfully enabled. 2023-02-12 03:59:15,375 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=256, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676174351364 in 386 msec 2023-02-12 03:59:15,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-12 03:59:15,745 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676174351364, procId: 256 completed 2023-02-12 03:59:15,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:59:15,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42205] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:59:15,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:59:15,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35785] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:59:15,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:59:15,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35783] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 03:59:15,761 DEBUG [Listener at localhost.localdomain/46483] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676174351364 2023-02-12 03:59:15,761 DEBUG [Listener at localhost.localdomain/46483] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:15,761 INFO [Listener at localhost.localdomain/46483] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-12 03:59:15,931 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-12 03:59:16,016 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-12 03:59:16,042 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-12 03:59:16,145 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-12 03:59:16,280 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-12 03:59:16,352 DEBUG [Listener at localhost.localdomain/46483] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-12 03:59:17,307 ERROR [VerifySnapshot-pool-0] snapshot.SnapshotReferenceUtil(301): Can't find hfile: a87ff679a2f3e71d9181a67b7542122c202302129ce77907bf8a43f6a2bfebfea85137d0 in the real (hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/a87ff679a2f3e71d9181a67b7542122c202302129ce77907bf8a43f6a2bfebfea85137d0) or archive (hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/a87ff679a2f3e71d9181a67b7542122c202302129ce77907bf8a43f6a2bfebfea85137d0) directory for the primary table. 2023-02-12 03:59:17,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: a87ff679a2f3e71d9181a67b7542122c202302129ce77907bf8a43f6a2bfebfea85137d0 in the real (hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/a87ff679a2f3e71d9181a67b7542122c202302129ce77907bf8a43f6a2bfebfea85137d0) or archive (hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/a87ff679a2f3e71d9181a67b7542122c202302129ce77907bf8a43f6a2bfebfea85137d0) directory for the primary table. at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:225) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:187) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.verifySnapshot(SnapshotReferenceUtil.java:164) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreOrCloneSnapshot(SnapshotManager.java:827) at org.apache.hadoop.hbase.master.HMaster$14.run(HMaster.java:2493) at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:132) at org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2489) at org.apache.hadoop.hbase.master.MasterRpcServices.restoreSnapshot(MasterRpcServices.java:1469) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 03:59:17,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] ipc.CallRunner(144): callId: 374 service: MasterService methodName: RestoreSnapshot size: 115 connection: 148.251.75.209:49656 deadline: 1676174417296, exception=org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: a87ff679a2f3e71d9181a67b7542122c202302129ce77907bf8a43f6a2bfebfea85137d0 in the real (hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/a87ff679a2f3e71d9181a67b7542122c202302129ce77907bf8a43f6a2bfebfea85137d0) or archive (hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/a87ff679a2f3e71d9181a67b7542122c202302129ce77907bf8a43f6a2bfebfea85137d0) directory for the primary table. 2023-02-12 03:59:17,312 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676174351364 2023-02-12 03:59:17,312 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$11(2405): Client=jenkins//148.251.75.209 disable testCorruptedSnapshot-1676174351364 2023-02-12 03:59:17,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=270, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676174351364 2023-02-12 03:59:17,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-12 03:59:17,315 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676174351364","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174357315"}]},"ts":"1676174357315"} 2023-02-12 03:59:17,316 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676174351364, state=DISABLING in hbase:meta 2023-02-12 03:59:17,317 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676174351364 to state=DISABLING 2023-02-12 03:59:17,318 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=ac12d10f8881ef03c212b7559a1cb27b, UNASSIGN}, {pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=1991ab242a39a7de82ac275617721040, UNASSIGN}, {pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=8d6f7fc2ff29d0e8082a3acf6b72d460, UNASSIGN}, {pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=395304cb23b0d8ab23e3ba3dc259249d, UNASSIGN}, {pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=5e163755ca31807434b0e014e67c9f89, UNASSIGN}, {pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=35bdba3092d7a6729f7490917901687d, UNASSIGN}] 2023-02-12 03:59:17,318 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=395304cb23b0d8ab23e3ba3dc259249d, UNASSIGN 2023-02-12 03:59:17,318 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=5e163755ca31807434b0e014e67c9f89, UNASSIGN 2023-02-12 03:59:17,319 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=8d6f7fc2ff29d0e8082a3acf6b72d460, UNASSIGN 2023-02-12 03:59:17,319 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=1991ab242a39a7de82ac275617721040, UNASSIGN 2023-02-12 03:59:17,319 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=35bdba3092d7a6729f7490917901687d, UNASSIGN 2023-02-12 03:59:17,319 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=8d6f7fc2ff29d0e8082a3acf6b72d460, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:17,319 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=395304cb23b0d8ab23e3ba3dc259249d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:17,319 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=5e163755ca31807434b0e014e67c9f89, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:17,319 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=1991ab242a39a7de82ac275617721040, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:17,319 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174357319"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174357319"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174357319"}]},"ts":"1676174357319"} 2023-02-12 03:59:17,320 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174357319"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174357319"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174357319"}]},"ts":"1676174357319"} 2023-02-12 03:59:17,319 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=35bdba3092d7a6729f7490917901687d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:17,319 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174357319"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174357319"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174357319"}]},"ts":"1676174357319"} 2023-02-12 03:59:17,319 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174357319"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174357319"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174357319"}]},"ts":"1676174357319"} 2023-02-12 03:59:17,320 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174357319"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174357319"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174357319"}]},"ts":"1676174357319"} 2023-02-12 03:59:17,320 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=277, ppid=272, state=RUNNABLE; CloseRegionProcedure 1991ab242a39a7de82ac275617721040, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:59:17,321 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=278, ppid=275, state=RUNNABLE; CloseRegionProcedure 5e163755ca31807434b0e014e67c9f89, server=jenkins-hbase20.apache.org,35783,1676174315451}] 2023-02-12 03:59:17,321 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=279, ppid=274, state=RUNNABLE; CloseRegionProcedure 395304cb23b0d8ab23e3ba3dc259249d, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:59:17,323 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=ac12d10f8881ef03c212b7559a1cb27b, UNASSIGN 2023-02-12 03:59:17,324 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=ac12d10f8881ef03c212b7559a1cb27b, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:17,324 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174357324"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1676174357324"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676174357324"}]},"ts":"1676174357324"} 2023-02-12 03:59:17,326 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=280, ppid=273, state=RUNNABLE; CloseRegionProcedure 8d6f7fc2ff29d0e8082a3acf6b72d460, server=jenkins-hbase20.apache.org,42205,1676174315413}] 2023-02-12 03:59:17,327 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=281, ppid=276, state=RUNNABLE; CloseRegionProcedure 35bdba3092d7a6729f7490917901687d, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:59:17,328 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=282, ppid=271, state=RUNNABLE; CloseRegionProcedure ac12d10f8881ef03c212b7559a1cb27b, server=jenkins-hbase20.apache.org,35785,1676174315481}] 2023-02-12 03:59:17,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:17,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 5e163755ca31807434b0e014e67c9f89, disabling compactions & flushes 2023-02-12 03:59:17,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:17,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:17,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. after waiting 0 ms 2023-02-12 03:59:17,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:17,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 5e163755ca31807434b0e014e67c9f89 1/1 column families, dataSize=2.29 KB heapSize=5.17 KB 2023-02-12 03:59:17,474 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:17,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 395304cb23b0d8ab23e3ba3dc259249d, disabling compactions & flushes 2023-02-12 03:59:17,474 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:17,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:17,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. after waiting 0 ms 2023-02-12 03:59:17,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:17,475 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 395304cb23b0d8ab23e3ba3dc259249d 1/1 column families, dataSize=1.96 KB heapSize=4.47 KB 2023-02-12 03:59:17,478 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:17,478 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 35bdba3092d7a6729f7490917901687d, disabling compactions & flushes 2023-02-12 03:59:17,478 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:17,478 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:17,478 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. after waiting 0 ms 2023-02-12 03:59:17,478 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:17,479 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 35bdba3092d7a6729f7490917901687d 1/1 column families, dataSize=23.22 KB heapSize=50.17 KB 2023-02-12 03:59:17,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf32023021236e2fe1c0db64e55b60954ef691e8b1d to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021236e2fe1c0db64e55b60954ef691e8b1d 2023-02-12 03:59:17,520 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230212cca809ce2b924aa09a8ad897141b247c to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/a87ff679a2f3e71d9181a67b7542122c20230212cca809ce2b924aa09a8ad897141b247c 2023-02-12 03:59:17,536 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d52023021210c550e6f4634c6aac047e4d706c1224 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/e4da3b7fbbce2345d7772b0674a318d52023021210c550e6f4634c6aac047e4d706c1224 2023-02-12 03:59:17,554 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/.tmp/cf/e6c0ada9e89645808ea8a15877653a9d 2023-02-12 03:59:17,557 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/.tmp/cf/e6c0ada9e89645808ea8a15877653a9d as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/cf/e6c0ada9e89645808ea8a15877653a9d 2023-02-12 03:59:17,561 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/cf/e6c0ada9e89645808ea8a15877653a9d, entries=35, sequenceid=15, filesize=10.9 K 2023-02-12 03:59:17,562 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.29 KB/2343, heapSize ~5.16 KB/5280, currentSize=0 B/0 for 5e163755ca31807434b0e014e67c9f89 in 89ms, sequenceid=15, compaction requested=false 2023-02-12 03:59:17,563 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/.tmp/cf/00deec2d18264c81a1c7b14422ff3381 2023-02-12 03:59:17,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/.tmp/cf/00deec2d18264c81a1c7b14422ff3381 as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/cf/00deec2d18264c81a1c7b14422ff3381 2023-02-12 03:59:17,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-12 03:59:17,568 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=23.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/.tmp/cf/fe06b487cc6e41e0abfc0c1548038959 2023-02-12 03:59:17,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 03:59:17,570 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. 2023-02-12 03:59:17,570 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 5e163755ca31807434b0e014e67c9f89: Waiting for close lock at 1676174357473Running coprocessor pre-close hooks at 1676174357473Disabling compacts and flushes for region at 1676174357473Disabling writes for close at 1676174357473Obtaining lock to block concurrent updates at 1676174357473Preparing flush snapshotting stores in 5e163755ca31807434b0e014e67c9f89 at 1676174357473Finished memstore snapshotting testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89., syncing WAL and waiting on mvcc, flushsize=dataSize=2343, getHeapSize=5280, getOffHeapSize=0, getCellsCount=35 at 1676174357474 (+1 ms)Flushing stores of testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89. at 1676174357474Flushing 5e163755ca31807434b0e014e67c9f89/cf: creating writer at 1676174357475 (+1 ms)Flushing mob file 5e163755ca31807434b0e014e67c9f89/cf: appending metadata at 1676174357482 (+7 ms)Flushing mob file 5e163755ca31807434b0e014e67c9f89/cf: closing flushed file at 1676174357482Flushing 5e163755ca31807434b0e014e67c9f89/cf: appending metadata at 1676174357521 (+39 ms)Flushing 5e163755ca31807434b0e014e67c9f89/cf: closing flushed file at 1676174357521Flushing 5e163755ca31807434b0e014e67c9f89/cf: reopening flushed file at 1676174357558 (+37 ms)Finished flush of dataSize ~2.29 KB/2343, heapSize ~5.16 KB/5280, currentSize=0 B/0 for 5e163755ca31807434b0e014e67c9f89 in 89ms, sequenceid=15, compaction requested=false at 1676174357562 (+4 ms)Writing region close event to WAL at 1676174357566 (+4 ms)Running coprocessor post-close hooks at 1676174357569 (+3 ms)Closed at 1676174357570 (+1 ms) 2023-02-12 03:59:17,572 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:17,572 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 1991ab242a39a7de82ac275617721040 2023-02-12 03:59:17,572 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 1991ab242a39a7de82ac275617721040, disabling compactions & flushes 2023-02-12 03:59:17,572 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:17,572 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:17,572 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. after waiting 0 ms 2023-02-12 03:59:17,572 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=5e163755ca31807434b0e014e67c9f89, regionState=CLOSED 2023-02-12 03:59:17,572 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:17,572 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174357572"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174357572"}]},"ts":"1676174357572"} 2023-02-12 03:59:17,572 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 1991ab242a39a7de82ac275617721040 1/1 column families, dataSize=1.90 KB heapSize=4.33 KB 2023-02-12 03:59:17,574 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/cf/00deec2d18264c81a1c7b14422ff3381, entries=30, sequenceid=15, filesize=10.1 K 2023-02-12 03:59:17,577 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/.tmp/cf/fe06b487cc6e41e0abfc0c1548038959 as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/cf/fe06b487cc6e41e0abfc0c1548038959 2023-02-12 03:59:17,577 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for 395304cb23b0d8ab23e3ba3dc259249d in 102ms, sequenceid=15, compaction requested=false 2023-02-12 03:59:17,578 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=278, resume processing ppid=275 2023-02-12 03:59:17,579 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=278, ppid=275, state=SUCCESS; CloseRegionProcedure 5e163755ca31807434b0e014e67c9f89, server=jenkins-hbase20.apache.org,35783,1676174315451 in 253 msec 2023-02-12 03:59:17,579 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=275, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=5e163755ca31807434b0e014e67c9f89, UNASSIGN in 260 msec 2023-02-12 03:59:17,581 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 03:59:17,581 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/cf/fe06b487cc6e41e0abfc0c1548038959, entries=355, sequenceid=15, filesize=66.1 K 2023-02-12 03:59:17,582 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. 2023-02-12 03:59:17,583 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 395304cb23b0d8ab23e3ba3dc259249d: Waiting for close lock at 1676174357474Running coprocessor pre-close hooks at 1676174357474Disabling compacts and flushes for region at 1676174357474Disabling writes for close at 1676174357475 (+1 ms)Obtaining lock to block concurrent updates at 1676174357475Preparing flush snapshotting stores in 395304cb23b0d8ab23e3ba3dc259249d at 1676174357475Finished memstore snapshotting testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d., syncing WAL and waiting on mvcc, flushsize=dataSize=2008, getHeapSize=4560, getOffHeapSize=0, getCellsCount=30 at 1676174357476 (+1 ms)Flushing stores of testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d. at 1676174357476Flushing 395304cb23b0d8ab23e3ba3dc259249d/cf: creating writer at 1676174357477 (+1 ms)Flushing mob file 395304cb23b0d8ab23e3ba3dc259249d/cf: appending metadata at 1676174357482 (+5 ms)Flushing mob file 395304cb23b0d8ab23e3ba3dc259249d/cf: closing flushed file at 1676174357482Flushing 395304cb23b0d8ab23e3ba3dc259249d/cf: appending metadata at 1676174357520 (+38 ms)Flushing 395304cb23b0d8ab23e3ba3dc259249d/cf: closing flushed file at 1676174357520Flushing 395304cb23b0d8ab23e3ba3dc259249d/cf: reopening flushed file at 1676174357567 (+47 ms)Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for 395304cb23b0d8ab23e3ba3dc259249d in 102ms, sequenceid=15, compaction requested=false at 1676174357577 (+10 ms)Writing region close event to WAL at 1676174357578 (+1 ms)Running coprocessor post-close hooks at 1676174357581 (+3 ms)Closed at 1676174357582 (+1 ms) 2023-02-12 03:59:17,583 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~23.22 KB/23781, heapSize ~50.16 KB/51360, currentSize=0 B/0 for 35bdba3092d7a6729f7490917901687d in 105ms, sequenceid=15, compaction requested=false 2023-02-12 03:59:17,584 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:17,584 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:17,584 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 8d6f7fc2ff29d0e8082a3acf6b72d460, disabling compactions & flushes 2023-02-12 03:59:17,584 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=395304cb23b0d8ab23e3ba3dc259249d, regionState=CLOSED 2023-02-12 03:59:17,584 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:17,584 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:17,584 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174357584"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174357584"}]},"ts":"1676174357584"} 2023-02-12 03:59:17,584 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. after waiting 0 ms 2023-02-12 03:59:17,584 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:17,584 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 8d6f7fc2ff29d0e8082a3acf6b72d460 1/1 column families, dataSize=1.83 KB heapSize=4.19 KB 2023-02-12 03:59:17,586 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=279, resume processing ppid=274 2023-02-12 03:59:17,586 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=279, ppid=274, state=SUCCESS; CloseRegionProcedure 395304cb23b0d8ab23e3ba3dc259249d, server=jenkins-hbase20.apache.org,42205,1676174315413 in 264 msec 2023-02-12 03:59:17,592 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=274, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=395304cb23b0d8ab23e3ba3dc259249d, UNASSIGN in 268 msec 2023-02-12 03:59:17,600 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 03:59:17,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. 2023-02-12 03:59:17,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 35bdba3092d7a6729f7490917901687d: Waiting for close lock at 1676174357478Running coprocessor pre-close hooks at 1676174357478Disabling compacts and flushes for region at 1676174357478Disabling writes for close at 1676174357478Obtaining lock to block concurrent updates at 1676174357479 (+1 ms)Preparing flush snapshotting stores in 35bdba3092d7a6729f7490917901687d at 1676174357479Finished memstore snapshotting testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d., syncing WAL and waiting on mvcc, flushsize=dataSize=23781, getHeapSize=51360, getOffHeapSize=0, getCellsCount=355 at 1676174357479Flushing stores of testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d. at 1676174357479Flushing 35bdba3092d7a6729f7490917901687d/cf: creating writer at 1676174357480 (+1 ms)Flushing mob file 35bdba3092d7a6729f7490917901687d/cf: appending metadata at 1676174357497 (+17 ms)Flushing mob file 35bdba3092d7a6729f7490917901687d/cf: closing flushed file at 1676174357497Flushing 35bdba3092d7a6729f7490917901687d/cf: appending metadata at 1676174357537 (+40 ms)Flushing 35bdba3092d7a6729f7490917901687d/cf: closing flushed file at 1676174357537Flushing 35bdba3092d7a6729f7490917901687d/cf: reopening flushed file at 1676174357578 (+41 ms)Finished flush of dataSize ~23.22 KB/23781, heapSize ~50.16 KB/51360, currentSize=0 B/0 for 35bdba3092d7a6729f7490917901687d in 105ms, sequenceid=15, compaction requested=false at 1676174357583 (+5 ms)Writing region close event to WAL at 1676174357591 (+8 ms)Running coprocessor post-close hooks at 1676174357601 (+10 ms)Closed at 1676174357602 (+1 ms) 2023-02-12 03:59:17,603 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:17,603 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:17,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing ac12d10f8881ef03c212b7559a1cb27b, disabling compactions & flushes 2023-02-12 03:59:17,604 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:17,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:17,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. after waiting 0 ms 2023-02-12 03:59:17,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:17,604 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing ac12d10f8881ef03c212b7559a1cb27b 1/1 column families, dataSize=1.50 KB heapSize=3.48 KB 2023-02-12 03:59:17,604 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=35bdba3092d7a6729f7490917901687d, regionState=CLOSED 2023-02-12 03:59:17,605 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174357604"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174357604"}]},"ts":"1676174357604"} 2023-02-12 03:59:17,618 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b2023021207197b3383c04af9b3883da677980a58 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/c4ca4238a0b923820dcc509a6f75849b2023021207197b3383c04af9b3883da677980a58 2023-02-12 03:59:17,623 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=281, resume processing ppid=276 2023-02-12 03:59:17,623 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=281, ppid=276, state=SUCCESS; CloseRegionProcedure 35bdba3092d7a6729f7490917901687d, server=jenkins-hbase20.apache.org,35785,1676174315481 in 285 msec 2023-02-12 03:59:17,628 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=276, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=35bdba3092d7a6729f7490917901687d, UNASSIGN in 305 msec 2023-02-12 03:59:17,634 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230212bef6a33a50df454ca86241ece6f9e55c to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/c81e728d9d4c2f636f067f89cc14862c20230212bef6a33a50df454ca86241ece6f9e55c 2023-02-12 03:59:17,663 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/.tmp/cf/15d3ba00c63341c5809e0004e9e5f67c 2023-02-12 03:59:17,666 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/.tmp/cf/15d3ba00c63341c5809e0004e9e5f67c as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/cf/15d3ba00c63341c5809e0004e9e5f67c 2023-02-12 03:59:17,669 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/cf/15d3ba00c63341c5809e0004e9e5f67c, entries=29, sequenceid=15, filesize=9.9 K 2023-02-12 03:59:17,670 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 1991ab242a39a7de82ac275617721040 in 98ms, sequenceid=15, compaction requested=false 2023-02-12 03:59:17,679 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202302124917942f83bb48d7acf4abd292f8e59b to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/d41d8cd98f00b204e9800998ecf8427e202302124917942f83bb48d7acf4abd292f8e59b 2023-02-12 03:59:17,679 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/.tmp/cf/a78172e87e6f4528941c0e71ad40aaa0 2023-02-12 03:59:17,681 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 03:59:17,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. 2023-02-12 03:59:17,722 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 1991ab242a39a7de82ac275617721040: Waiting for close lock at 1676174357572Running coprocessor pre-close hooks at 1676174357572Disabling compacts and flushes for region at 1676174357572Disabling writes for close at 1676174357572Obtaining lock to block concurrent updates at 1676174357572Preparing flush snapshotting stores in 1991ab242a39a7de82ac275617721040 at 1676174357572Finished memstore snapshotting testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040., syncing WAL and waiting on mvcc, flushsize=dataSize=1941, getHeapSize=4416, getOffHeapSize=0, getCellsCount=29 at 1676174357572Flushing stores of testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040. at 1676174357573 (+1 ms)Flushing 1991ab242a39a7de82ac275617721040/cf: creating writer at 1676174357574 (+1 ms)Flushing mob file 1991ab242a39a7de82ac275617721040/cf: appending metadata at 1676174357580 (+6 ms)Flushing mob file 1991ab242a39a7de82ac275617721040/cf: closing flushed file at 1676174357580Flushing 1991ab242a39a7de82ac275617721040/cf: appending metadata at 1676174357619 (+39 ms)Flushing 1991ab242a39a7de82ac275617721040/cf: closing flushed file at 1676174357619Flushing 1991ab242a39a7de82ac275617721040/cf: reopening flushed file at 1676174357667 (+48 ms)Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 1991ab242a39a7de82ac275617721040 in 98ms, sequenceid=15, compaction requested=false at 1676174357670 (+3 ms)Writing region close event to WAL at 1676174357676 (+6 ms)Running coprocessor post-close hooks at 1676174357682 (+6 ms)Closed at 1676174357683 (+1 ms) 2023-02-12 03:59:17,723 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x79efc34c to 127.0.0.1:50368 2023-02-12 03:59:17,723 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:59:17,724 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 1991ab242a39a7de82ac275617721040 2023-02-12 03:59:17,724 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=1991ab242a39a7de82ac275617721040, regionState=CLOSED 2023-02-12 03:59:17,725 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174357724"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174357724"}]},"ts":"1676174357724"} 2023-02-12 03:59:17,727 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=277, resume processing ppid=272 2023-02-12 03:59:17,727 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=277, ppid=272, state=SUCCESS; CloseRegionProcedure 1991ab242a39a7de82ac275617721040, server=jenkins-hbase20.apache.org,35783,1676174315451 in 406 msec 2023-02-12 03:59:17,728 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=272, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=1991ab242a39a7de82ac275617721040, UNASSIGN in 409 msec 2023-02-12 03:59:17,734 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/.tmp/cf/a78172e87e6f4528941c0e71ad40aaa0 as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/cf/a78172e87e6f4528941c0e71ad40aaa0 2023-02-12 03:59:17,737 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/cf/a78172e87e6f4528941c0e71ad40aaa0, entries=28, sequenceid=15, filesize=9.7 K 2023-02-12 03:59:17,738 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for 8d6f7fc2ff29d0e8082a3acf6b72d460 in 154ms, sequenceid=15, compaction requested=false 2023-02-12 03:59:17,746 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/.tmp/cf/14868aa13d864a5085826d5c7cbf3571 2023-02-12 03:59:17,749 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 03:59:17,752 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. 2023-02-12 03:59:17,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/.tmp/cf/14868aa13d864a5085826d5c7cbf3571 as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/cf/14868aa13d864a5085826d5c7cbf3571 2023-02-12 03:59:17,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 8d6f7fc2ff29d0e8082a3acf6b72d460: Waiting for close lock at 1676174357584Running coprocessor pre-close hooks at 1676174357584Disabling compacts and flushes for region at 1676174357584Disabling writes for close at 1676174357584Obtaining lock to block concurrent updates at 1676174357584Preparing flush snapshotting stores in 8d6f7fc2ff29d0e8082a3acf6b72d460 at 1676174357584Finished memstore snapshotting testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460., syncing WAL and waiting on mvcc, flushsize=dataSize=1874, getHeapSize=4272, getOffHeapSize=0, getCellsCount=28 at 1676174357585 (+1 ms)Flushing stores of testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460. at 1676174357585Flushing 8d6f7fc2ff29d0e8082a3acf6b72d460/cf: creating writer at 1676174357588 (+3 ms)Flushing mob file 8d6f7fc2ff29d0e8082a3acf6b72d460/cf: appending metadata at 1676174357599 (+11 ms)Flushing mob file 8d6f7fc2ff29d0e8082a3acf6b72d460/cf: closing flushed file at 1676174357599Flushing 8d6f7fc2ff29d0e8082a3acf6b72d460/cf: appending metadata at 1676174357635 (+36 ms)Flushing 8d6f7fc2ff29d0e8082a3acf6b72d460/cf: closing flushed file at 1676174357635Flushing 8d6f7fc2ff29d0e8082a3acf6b72d460/cf: reopening flushed file at 1676174357735 (+100 ms)Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for 8d6f7fc2ff29d0e8082a3acf6b72d460 in 154ms, sequenceid=15, compaction requested=false at 1676174357738 (+3 ms)Writing region close event to WAL at 1676174357746 (+8 ms)Running coprocessor post-close hooks at 1676174357751 (+5 ms)Closed at 1676174357752 (+1 ms) 2023-02-12 03:59:17,754 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:17,754 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=8d6f7fc2ff29d0e8082a3acf6b72d460, regionState=CLOSED 2023-02-12 03:59:17,754 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676174357754"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174357754"}]},"ts":"1676174357754"} 2023-02-12 03:59:17,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/cf/14868aa13d864a5085826d5c7cbf3571, entries=23, sequenceid=15, filesize=8.8 K 2023-02-12 03:59:17,756 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.50 KB/1539, heapSize ~3.47 KB/3552, currentSize=0 B/0 for ac12d10f8881ef03c212b7559a1cb27b in 152ms, sequenceid=15, compaction requested=false 2023-02-12 03:59:17,757 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=280, resume processing ppid=273 2023-02-12 03:59:17,757 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=280, ppid=273, state=SUCCESS; CloseRegionProcedure 8d6f7fc2ff29d0e8082a3acf6b72d460, server=jenkins-hbase20.apache.org,42205,1676174315413 in 430 msec 2023-02-12 03:59:17,758 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=273, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=8d6f7fc2ff29d0e8082a3acf6b72d460, UNASSIGN in 439 msec 2023-02-12 03:59:17,759 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 03:59:17,761 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. 2023-02-12 03:59:17,761 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for ac12d10f8881ef03c212b7559a1cb27b: Waiting for close lock at 1676174357603Running coprocessor pre-close hooks at 1676174357603Disabling compacts and flushes for region at 1676174357603Disabling writes for close at 1676174357604 (+1 ms)Obtaining lock to block concurrent updates at 1676174357604Preparing flush snapshotting stores in ac12d10f8881ef03c212b7559a1cb27b at 1676174357604Finished memstore snapshotting testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b., syncing WAL and waiting on mvcc, flushsize=dataSize=1539, getHeapSize=3552, getOffHeapSize=0, getCellsCount=23 at 1676174357604Flushing stores of testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b. at 1676174357605 (+1 ms)Flushing ac12d10f8881ef03c212b7559a1cb27b/cf: creating writer at 1676174357611 (+6 ms)Flushing mob file ac12d10f8881ef03c212b7559a1cb27b/cf: appending metadata at 1676174357631 (+20 ms)Flushing mob file ac12d10f8881ef03c212b7559a1cb27b/cf: closing flushed file at 1676174357631Flushing ac12d10f8881ef03c212b7559a1cb27b/cf: appending metadata at 1676174357680 (+49 ms)Flushing ac12d10f8881ef03c212b7559a1cb27b/cf: closing flushed file at 1676174357680Flushing ac12d10f8881ef03c212b7559a1cb27b/cf: reopening flushed file at 1676174357752 (+72 ms)Finished flush of dataSize ~1.50 KB/1539, heapSize ~3.47 KB/3552, currentSize=0 B/0 for ac12d10f8881ef03c212b7559a1cb27b in 152ms, sequenceid=15, compaction requested=false at 1676174357756 (+4 ms)Writing region close event to WAL at 1676174357756Running coprocessor post-close hooks at 1676174357760 (+4 ms)Closed at 1676174357761 (+1 ms) 2023-02-12 03:59:17,762 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:17,762 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=ac12d10f8881ef03c212b7559a1cb27b, regionState=CLOSED 2023-02-12 03:59:17,762 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676174357762"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676174357762"}]},"ts":"1676174357762"} 2023-02-12 03:59:17,765 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=282, resume processing ppid=271 2023-02-12 03:59:17,765 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=282, ppid=271, state=SUCCESS; CloseRegionProcedure ac12d10f8881ef03c212b7559a1cb27b, server=jenkins-hbase20.apache.org,35785,1676174315481 in 435 msec 2023-02-12 03:59:17,766 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=271, resume processing ppid=270 2023-02-12 03:59:17,766 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=271, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676174351364, region=ac12d10f8881ef03c212b7559a1cb27b, UNASSIGN in 447 msec 2023-02-12 03:59:17,766 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676174351364","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676174357766"}]},"ts":"1676174357766"} 2023-02-12 03:59:17,767 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676174351364, state=DISABLED in hbase:meta 2023-02-12 03:59:17,825 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676174351364 to state=DISABLED 2023-02-12 03:59:17,831 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=270, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676174351364 in 514 msec 2023-02-12 03:59:18,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-12 03:59:18,067 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676174351364, procId: 270 completed 2023-02-12 03:59:18,068 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.HMaster$5(2215): Client=jenkins//148.251.75.209 delete testCorruptedSnapshot-1676174351364 2023-02-12 03:59:18,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] procedure2.ProcedureExecutor(1029): Stored pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testCorruptedSnapshot-1676174351364 2023-02-12 03:59:18,070 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676174351364 2023-02-12 03:59:18,071 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=283, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676174351364 2023-02-12 03:59:18,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-12 03:59:18,077 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:18,077 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:18,077 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:18,077 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:18,077 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:18,077 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040 2023-02-12 03:59:18,080 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/cf, FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/recovered.edits] 2023-02-12 03:59:18,080 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/cf, FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/recovered.edits] 2023-02-12 03:59:18,080 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/cf, FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/recovered.edits] 2023-02-12 03:59:18,081 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/cf, FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/recovered.edits] 2023-02-12 03:59:18,081 DEBUG [HFileArchiver-10] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/cf, FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/recovered.edits] 2023-02-12 03:59:18,082 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/cf, FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/recovered.edits] 2023-02-12 03:59:18,086 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/cf/002ae5f09d194b26bbdce801282de0b5 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/cf/002ae5f09d194b26bbdce801282de0b5 2023-02-12 03:59:18,087 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/cf/ae19421e9ec14c2c91a5a16c0bfd026e to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/cf/ae19421e9ec14c2c91a5a16c0bfd026e 2023-02-12 03:59:18,087 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/cf/107ee1abf95441bb87814c20220acd44 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/cf/107ee1abf95441bb87814c20220acd44 2023-02-12 03:59:18,088 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/cf/00deec2d18264c81a1c7b14422ff3381 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/cf/00deec2d18264c81a1c7b14422ff3381 2023-02-12 03:59:18,088 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/cf/14868aa13d864a5085826d5c7cbf3571 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/cf/14868aa13d864a5085826d5c7cbf3571 2023-02-12 03:59:18,089 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/cf/e58e4a3220514ebaa33fd06e0beba2b6 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/cf/e58e4a3220514ebaa33fd06e0beba2b6 2023-02-12 03:59:18,089 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/cf/15d3ba00c63341c5809e0004e9e5f67c to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/cf/15d3ba00c63341c5809e0004e9e5f67c 2023-02-12 03:59:18,089 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/cf/e6c0ada9e89645808ea8a15877653a9d to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/cf/e6c0ada9e89645808ea8a15877653a9d 2023-02-12 03:59:18,090 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/cf/044a1a8620ec494e808ac6dcfe07f34f to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/cf/044a1a8620ec494e808ac6dcfe07f34f 2023-02-12 03:59:18,090 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/cf/a78172e87e6f4528941c0e71ad40aaa0 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/cf/a78172e87e6f4528941c0e71ad40aaa0 2023-02-12 03:59:18,092 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/cf/c203a1de813c4f6fa4f8547870b13bea to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/cf/c203a1de813c4f6fa4f8547870b13bea 2023-02-12 03:59:18,092 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/cf/fe06b487cc6e41e0abfc0c1548038959 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/cf/fe06b487cc6e41e0abfc0c1548038959 2023-02-12 03:59:18,100 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/recovered.edits/18.seqid to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040/recovered.edits/18.seqid 2023-02-12 03:59:18,100 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/recovered.edits/18.seqid to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b/recovered.edits/18.seqid 2023-02-12 03:59:18,100 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/recovered.edits/18.seqid to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89/recovered.edits/18.seqid 2023-02-12 03:59:18,100 DEBUG [HFileArchiver-10] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/1991ab242a39a7de82ac275617721040 2023-02-12 03:59:18,101 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/recovered.edits/18.seqid to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460/recovered.edits/18.seqid 2023-02-12 03:59:18,101 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/recovered.edits/18.seqid to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d/recovered.edits/18.seqid 2023-02-12 03:59:18,101 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/ac12d10f8881ef03c212b7559a1cb27b 2023-02-12 03:59:18,103 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/5e163755ca31807434b0e014e67c9f89 2023-02-12 03:59:18,103 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/recovered.edits/18.seqid to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d/recovered.edits/18.seqid 2023-02-12 03:59:18,103 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/395304cb23b0d8ab23e3ba3dc259249d 2023-02-12 03:59:18,103 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/8d6f7fc2ff29d0e8082a3acf6b72d460 2023-02-12 03:59:18,104 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/.tmp/data/default/testCorruptedSnapshot-1676174351364/35bdba3092d7a6729f7490917901687d 2023-02-12 03:59:18,104 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676174351364 regions 2023-02-12 03:59:18,104 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918 2023-02-12 03:59:18,104 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf] 2023-02-12 03:59:18,107 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/a87ff679a2f3e71d9181a67b7542122c20230212cca809ce2b924aa09a8ad897141b247c to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/a87ff679a2f3e71d9181a67b7542122c20230212cca809ce2b924aa09a8ad897141b247c 2023-02-12 03:59:18,108 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/c4ca4238a0b923820dcc509a6f75849b2023021207197b3383c04af9b3883da677980a58 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/c4ca4238a0b923820dcc509a6f75849b2023021207197b3383c04af9b3883da677980a58 2023-02-12 03:59:18,109 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/c4ca4238a0b923820dcc509a6f75849b202302122fccebde9fc74f9e8335c7d37fc7a3bc to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/c4ca4238a0b923820dcc509a6f75849b202302122fccebde9fc74f9e8335c7d37fc7a3bc 2023-02-12 03:59:18,109 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/c81e728d9d4c2f636f067f89cc14862c2023021281504bda48ef46eba3abad932559abc7 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/c81e728d9d4c2f636f067f89cc14862c2023021281504bda48ef46eba3abad932559abc7 2023-02-12 03:59:18,110 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/c81e728d9d4c2f636f067f89cc14862c20230212bef6a33a50df454ca86241ece6f9e55c to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/c81e728d9d4c2f636f067f89cc14862c20230212bef6a33a50df454ca86241ece6f9e55c 2023-02-12 03:59:18,111 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/d41d8cd98f00b204e9800998ecf8427e202302124917942f83bb48d7acf4abd292f8e59b to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/d41d8cd98f00b204e9800998ecf8427e202302124917942f83bb48d7acf4abd292f8e59b 2023-02-12 03:59:18,112 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/d41d8cd98f00b204e9800998ecf8427e202302129ef383240168403db147f26b5bec256d to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/d41d8cd98f00b204e9800998ecf8427e202302129ef383240168403db147f26b5bec256d 2023-02-12 03:59:18,114 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/e4da3b7fbbce2345d7772b0674a318d52023021210c550e6f4634c6aac047e4d706c1224 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/e4da3b7fbbce2345d7772b0674a318d52023021210c550e6f4634c6aac047e4d706c1224 2023-02-12 03:59:18,117 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/e4da3b7fbbce2345d7772b0674a318d520230212474eeddee91647a29b4bb8010100f7a3 to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/e4da3b7fbbce2345d7772b0674a318d520230212474eeddee91647a29b4bb8010100f7a3 2023-02-12 03:59:18,118 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021236e2fe1c0db64e55b60954ef691e8b1d to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021236e2fe1c0db64e55b60954ef691e8b1d 2023-02-12 03:59:18,119 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212c291ab8dd9144f9583d27478af0d785a to hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/archive/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212c291ab8dd9144f9583d27478af0d785a 2023-02-12 03:59:18,120 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/mobdir/data/default/testCorruptedSnapshot-1676174351364/2e3162416757b1af395cf673faebb918 2023-02-12 03:59:18,121 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=283, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676174351364 2023-02-12 03:59:18,124 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testCorruptedSnapshot-1676174351364 from hbase:meta 2023-02-12 03:59:18,130 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'testCorruptedSnapshot-1676174351364' descriptor. 2023-02-12 03:59:18,131 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=283, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676174351364 2023-02-12 03:59:18,131 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'testCorruptedSnapshot-1676174351364' from region states. 2023-02-12 03:59:18,132 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174358131"}]},"ts":"9223372036854775807"} 2023-02-12 03:59:18,132 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174358131"}]},"ts":"9223372036854775807"} 2023-02-12 03:59:18,132 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174358131"}]},"ts":"9223372036854775807"} 2023-02-12 03:59:18,132 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174358131"}]},"ts":"9223372036854775807"} 2023-02-12 03:59:18,132 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174358131"}]},"ts":"9223372036854775807"} 2023-02-12 03:59:18,132 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676174358131"}]},"ts":"9223372036854775807"} 2023-02-12 03:59:18,134 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-12 03:59:18,134 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => ac12d10f8881ef03c212b7559a1cb27b, NAME => 'testCorruptedSnapshot-1676174351364,,1676174351365.ac12d10f8881ef03c212b7559a1cb27b.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 1991ab242a39a7de82ac275617721040, NAME => 'testCorruptedSnapshot-1676174351364,1,1676174351365.1991ab242a39a7de82ac275617721040.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 8d6f7fc2ff29d0e8082a3acf6b72d460, NAME => 'testCorruptedSnapshot-1676174351364,2,1676174351365.8d6f7fc2ff29d0e8082a3acf6b72d460.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 395304cb23b0d8ab23e3ba3dc259249d, NAME => 'testCorruptedSnapshot-1676174351364,3,1676174351365.395304cb23b0d8ab23e3ba3dc259249d.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 5e163755ca31807434b0e014e67c9f89, NAME => 'testCorruptedSnapshot-1676174351364,4,1676174351365.5e163755ca31807434b0e014e67c9f89.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 35bdba3092d7a6729f7490917901687d, NAME => 'testCorruptedSnapshot-1676174351364,5,1676174351365.35bdba3092d7a6729f7490917901687d.', STARTKEY => '5', ENDKEY => ''}] 2023-02-12 03:59:18,134 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'testCorruptedSnapshot-1676174351364' as deleted. 2023-02-12 03:59:18,134 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676174351364","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676174358134"}]},"ts":"9223372036854775807"} 2023-02-12 03:59:18,136 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table testCorruptedSnapshot-1676174351364 state from META 2023-02-12 03:59:18,138 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=283, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676174351364 2023-02-12 03:59:18,139 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=283, state=SUCCESS; DeleteTableProcedure table=testCorruptedSnapshot-1676174351364 in 70 msec 2023-02-12 03:59:18,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-12 03:59:18,323 INFO [Listener at localhost.localdomain/46483] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testCorruptedSnapshot-1676174351364, procId: 283 completed 2023-02-12 03:59:18,327 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(748): Client=jenkins//148.251.75.209 delete name: "emptySnaptb-1676174351364" 2023-02-12 03:59:18,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676174351364 2023-02-12 03:59:18,330 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] master.MasterRpcServices(748): Client=jenkins//148.251.75.209 delete name: "snaptb0-1676174351364" 2023-02-12 03:59:18,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37305] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676174351364 2023-02-12 03:59:18,351 INFO [Listener at localhost.localdomain/46483] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=420 (was 408) Potentially hanging thread: hconnection-0x51e75f37-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x51e75f37-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x51e75f37-shared-pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7a990b1c-shared-pool-34 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1710533965_15 at /127.0.0.1:39648 [Waiting for operation #777] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1207670672_15 at /127.0.0.1:40290 [Waiting for operation #278] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7a990b1c-shared-pool-35 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x51e75f37-shared-pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-463782719_15 at /127.0.0.1:39640 [Waiting for operation #527] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1207670672_15 at /127.0.0.1:47114 [Waiting for operation #742] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-463782719_15 at /127.0.0.1:42578 [Waiting for operation #265] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7a990b1c-shared-pool-36 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-891324994_15 at /127.0.0.1:47106 [Waiting for operation #774] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1376698933_15 at /127.0.0.1:47130 [Waiting for operation #11] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1207670672_15 at /127.0.0.1:40282 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1710533965_15 at /127.0.0.1:40284 [Waiting for operation #263] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-463782719_15 at /127.0.0.1:35502 [Waiting for operation #7] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-463782719_15 at /127.0.0.1:35210 [Waiting for operation #23] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7a990b1c-shared-pool-37 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x51e75f37-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1710533965_15 at /127.0.0.1:39804 [Waiting for operation #13] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=794 (was 786) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=295 (was 320), ProcessCount=164 (was 164), AvailableMemoryMB=3514 (was 3754) 2023-02-12 03:59:18,353 INFO [Listener at localhost.localdomain/46483] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-12 03:59:18,353 INFO [Listener at localhost.localdomain/46483] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-12 03:59:18,353 DEBUG [Listener at localhost.localdomain/46483] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d193e18 to 127.0.0.1:50368 2023-02-12 03:59:18,354 DEBUG [Listener at localhost.localdomain/46483] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:59:18,354 DEBUG [Listener at localhost.localdomain/46483] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-12 03:59:18,354 DEBUG [Listener at localhost.localdomain/46483] util.JVMClusterUtil(257): Found active master hash=2096480596, stopped=false 2023-02-12 03:59:18,354 INFO [Listener at localhost.localdomain/46483] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,37305,1676174314460 2023-02-12 03:59:18,356 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-12 03:59:18,356 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:35783-0x10154d60a810002, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-12 03:59:18,356 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:35785-0x10154d60a810003, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-12 03:59:18,356 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:42205-0x10154d60a810001, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-12 03:59:18,356 INFO [Listener at localhost.localdomain/46483] procedure2.ProcedureExecutor(629): Stopping 2023-02-12 03:59:18,356 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-12 03:59:18,356 DEBUG [Listener at localhost.localdomain/46483] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0df654e0 to 127.0.0.1:50368 2023-02-12 03:59:18,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35783-0x10154d60a810002, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-12 03:59:18,357 DEBUG [Listener at localhost.localdomain/46483] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:59:18,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-12 03:59:18,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35785-0x10154d60a810003, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-12 03:59:18,357 INFO [Listener at localhost.localdomain/46483] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,42205,1676174315413' ***** 2023-02-12 03:59:18,357 INFO [Listener at localhost.localdomain/46483] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-12 03:59:18,357 INFO [Listener at localhost.localdomain/46483] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,35783,1676174315451' ***** 2023-02-12 03:59:18,357 INFO [RS:0;jenkins-hbase20:42205] regionserver.HeapMemoryManager(220): Stopping 2023-02-12 03:59:18,357 INFO [Listener at localhost.localdomain/46483] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-12 03:59:18,358 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-12 03:59:18,358 INFO [Listener at localhost.localdomain/46483] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,35785,1676174315481' ***** 2023-02-12 03:59:18,358 INFO [Listener at localhost.localdomain/46483] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-12 03:59:18,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42205-0x10154d60a810001, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-12 03:59:18,358 INFO [RS:2;jenkins-hbase20:35785] regionserver.HeapMemoryManager(220): Stopping 2023-02-12 03:59:18,358 INFO [RS:1;jenkins-hbase20:35783] regionserver.HeapMemoryManager(220): Stopping 2023-02-12 03:59:18,358 INFO [RS:2;jenkins-hbase20:35785] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-12 03:59:18,358 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-12 03:59:18,358 INFO [RS:0;jenkins-hbase20:42205] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-12 03:59:18,358 INFO [RS:2;jenkins-hbase20:35785] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-12 03:59:18,358 INFO [RS:1;jenkins-hbase20:35783] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-12 03:59:18,359 INFO [RS:2;jenkins-hbase20:35785] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:18,358 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-12 03:59:18,359 DEBUG [RS:2;jenkins-hbase20:35785] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f6bc73c to 127.0.0.1:50368 2023-02-12 03:59:18,359 INFO [RS:1;jenkins-hbase20:35783] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-12 03:59:18,359 INFO [RS:1;jenkins-hbase20:35783] regionserver.HRegionServer(3304): Received CLOSE for d852011c2bc11fbb7a1e9bd27375aba0 2023-02-12 03:59:18,358 INFO [RS:0;jenkins-hbase20:42205] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-12 03:59:18,359 INFO [RS:0;jenkins-hbase20:42205] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:18,359 DEBUG [RS:0;jenkins-hbase20:42205] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36b06f89 to 127.0.0.1:50368 2023-02-12 03:59:18,359 DEBUG [RS:0;jenkins-hbase20:42205] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:59:18,359 INFO [RS:1;jenkins-hbase20:35783] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:18,359 DEBUG [RS:1;jenkins-hbase20:35783] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x491252d5 to 127.0.0.1:50368 2023-02-12 03:59:18,359 DEBUG [RS:2;jenkins-hbase20:35785] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:59:18,359 DEBUG [RS:1;jenkins-hbase20:35783] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:59:18,359 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing d852011c2bc11fbb7a1e9bd27375aba0, disabling compactions & flushes 2023-02-12 03:59:18,359 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0. 2023-02-12 03:59:18,360 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0. 2023-02-12 03:59:18,360 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0. after waiting 0 ms 2023-02-12 03:59:18,360 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0. 2023-02-12 03:59:18,359 INFO [RS:0;jenkins-hbase20:42205] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,42205,1676174315413; all regions closed. 2023-02-12 03:59:18,360 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing d852011c2bc11fbb7a1e9bd27375aba0 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-12 03:59:18,359 INFO [RS:1;jenkins-hbase20:35783] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-12 03:59:18,360 INFO [RS:1;jenkins-hbase20:35783] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-12 03:59:18,360 INFO [RS:1;jenkins-hbase20:35783] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-12 03:59:18,359 INFO [RS:2;jenkins-hbase20:35785] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,35785,1676174315481; all regions closed. 2023-02-12 03:59:18,360 INFO [RS:1;jenkins-hbase20:35783] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-12 03:59:18,361 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-12 03:59:18,361 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-12 03:59:18,361 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-12 03:59:18,361 INFO [RS:1;jenkins-hbase20:35783] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-12 03:59:18,362 DEBUG [RS:1;jenkins-hbase20:35783] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, d852011c2bc11fbb7a1e9bd27375aba0=hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0.} 2023-02-12 03:59:18,362 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-12 03:59:18,362 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-12 03:59:18,362 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-12 03:59:18,362 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-12 03:59:18,362 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-12 03:59:18,362 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=137.88 KB heapSize=212.75 KB 2023-02-12 03:59:18,363 DEBUG [RS:1;jenkins-hbase20:35783] regionserver.HRegionServer(1505): Waiting on 1588230740, d852011c2bc11fbb7a1e9bd27375aba0 2023-02-12 03:59:18,387 DEBUG [RS:2;jenkins-hbase20:35785] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/oldWALs 2023-02-12 03:59:18,387 DEBUG [RS:0;jenkins-hbase20:42205] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/oldWALs 2023-02-12 03:59:18,388 INFO [RS:0;jenkins-hbase20:42205] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C42205%2C1676174315413:(num 1676174316836) 2023-02-12 03:59:18,388 INFO [RS:2;jenkins-hbase20:35785] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C35785%2C1676174315481:(num 1676174316836) 2023-02-12 03:59:18,388 DEBUG [RS:2;jenkins-hbase20:35785] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:59:18,388 DEBUG [RS:0;jenkins-hbase20:42205] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:59:18,388 INFO [RS:2;jenkins-hbase20:35785] regionserver.LeaseManager(133): Closed leases 2023-02-12 03:59:18,388 INFO [RS:0;jenkins-hbase20:42205] regionserver.LeaseManager(133): Closed leases 2023-02-12 03:59:18,394 INFO [RS:0;jenkins-hbase20:42205] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-12 03:59:18,394 INFO [RS:2;jenkins-hbase20:35785] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-12 03:59:18,395 INFO [RS:0;jenkins-hbase20:42205] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-12 03:59:18,395 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-12 03:59:18,395 INFO [RS:0;jenkins-hbase20:42205] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-12 03:59:18,395 INFO [RS:0;jenkins-hbase20:42205] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-12 03:59:18,395 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-12 03:59:18,395 INFO [RS:2;jenkins-hbase20:35785] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-12 03:59:18,395 INFO [RS:2;jenkins-hbase20:35785] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-12 03:59:18,395 INFO [RS:2;jenkins-hbase20:35785] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-12 03:59:18,397 INFO [RS:0;jenkins-hbase20:42205] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:42205 2023-02-12 03:59:18,398 INFO [RS:2;jenkins-hbase20:35785] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:35785 2023-02-12 03:59:18,400 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/namespace/d852011c2bc11fbb7a1e9bd27375aba0/.tmp/info/8b49750ab7944ebbbad87bc26b1b947e 2023-02-12 03:59:18,403 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=131.12 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/.tmp/info/09ceff8713464fa99da7c98a88f5f37d 2023-02-12 03:59:18,403 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/namespace/d852011c2bc11fbb7a1e9bd27375aba0/.tmp/info/8b49750ab7944ebbbad87bc26b1b947e as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/namespace/d852011c2bc11fbb7a1e9bd27375aba0/info/8b49750ab7944ebbbad87bc26b1b947e 2023-02-12 03:59:18,406 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 09ceff8713464fa99da7c98a88f5f37d 2023-02-12 03:59:18,407 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/namespace/d852011c2bc11fbb7a1e9bd27375aba0/info/8b49750ab7944ebbbad87bc26b1b947e, entries=2, sequenceid=6, filesize=4.8 K 2023-02-12 03:59:18,408 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for d852011c2bc11fbb7a1e9bd27375aba0 in 48ms, sequenceid=6, compaction requested=false 2023-02-12 03:59:18,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/namespace/d852011c2bc11fbb7a1e9bd27375aba0/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-12 03:59:18,418 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0. 2023-02-12 03:59:18,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for d852011c2bc11fbb7a1e9bd27375aba0: Waiting for close lock at 1676174358359Running coprocessor pre-close hooks at 1676174358359Disabling compacts and flushes for region at 1676174358359Disabling writes for close at 1676174358360 (+1 ms)Obtaining lock to block concurrent updates at 1676174358360Preparing flush snapshotting stores in d852011c2bc11fbb7a1e9bd27375aba0 at 1676174358360Finished memstore snapshotting hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676174358360Flushing stores of hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0. at 1676174358361 (+1 ms)Flushing d852011c2bc11fbb7a1e9bd27375aba0/info: creating writer at 1676174358361Flushing d852011c2bc11fbb7a1e9bd27375aba0/info: appending metadata at 1676174358366 (+5 ms)Flushing d852011c2bc11fbb7a1e9bd27375aba0/info: closing flushed file at 1676174358366Flushing d852011c2bc11fbb7a1e9bd27375aba0/info: reopening flushed file at 1676174358404 (+38 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for d852011c2bc11fbb7a1e9bd27375aba0 in 48ms, sequenceid=6, compaction requested=false at 1676174358408 (+4 ms)Writing region close event to WAL at 1676174358409 (+1 ms)Running coprocessor post-close hooks at 1676174358415 (+6 ms)Closed at 1676174358418 (+3 ms) 2023-02-12 03:59:18,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676174317255.d852011c2bc11fbb7a1e9bd27375aba0. 2023-02-12 03:59:18,426 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.01 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/.tmp/rep_barrier/843a57ce21e2457cbf424ebb6cb65dbd 2023-02-12 03:59:18,428 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 843a57ce21e2457cbf424ebb6cb65dbd 2023-02-12 03:59:18,449 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.75 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/.tmp/table/a61f0c4967b34772a90bac0e6119b40f 2023-02-12 03:59:18,452 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a61f0c4967b34772a90bac0e6119b40f 2023-02-12 03:59:18,452 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/.tmp/info/09ceff8713464fa99da7c98a88f5f37d as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/info/09ceff8713464fa99da7c98a88f5f37d 2023-02-12 03:59:18,455 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 09ceff8713464fa99da7c98a88f5f37d 2023-02-12 03:59:18,455 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/info/09ceff8713464fa99da7c98a88f5f37d, entries=64, sequenceid=307, filesize=11.9 K 2023-02-12 03:59:18,456 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/.tmp/rep_barrier/843a57ce21e2457cbf424ebb6cb65dbd as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/rep_barrier/843a57ce21e2457cbf424ebb6cb65dbd 2023-02-12 03:59:18,458 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 843a57ce21e2457cbf424ebb6cb65dbd 2023-02-12 03:59:18,458 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/rep_barrier/843a57ce21e2457cbf424ebb6cb65dbd, entries=18, sequenceid=307, filesize=7.0 K 2023-02-12 03:59:18,459 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/.tmp/table/a61f0c4967b34772a90bac0e6119b40f as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/table/a61f0c4967b34772a90bac0e6119b40f 2023-02-12 03:59:18,463 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-12 03:59:18,463 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:35783-0x10154d60a810002, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:18,463 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:42205-0x10154d60a810001, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:18,463 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:35785-0x10154d60a810003, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,35785,1676174315481 2023-02-12 03:59:18,464 ERROR [Listener at localhost.localdomain/46483-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7ce6ee7 rejected from java.util.concurrent.ThreadPoolExecutor@3f1eac6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 03:59:18,464 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:42205-0x10154d60a810001, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-12 03:59:18,464 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:35783-0x10154d60a810002, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-12 03:59:18,465 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:35785-0x10154d60a810003, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-12 03:59:18,465 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:35783-0x10154d60a810002, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:18,465 ERROR [Listener at localhost.localdomain/46483-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3325985 rejected from java.util.concurrent.ThreadPoolExecutor@3f1eac6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 03:59:18,465 ERROR [Listener at localhost.localdomain/46483-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@752c401e rejected from java.util.concurrent.ThreadPoolExecutor@7113da63[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 03:59:18,465 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:35785-0x10154d60a810003, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:18,465 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:42205-0x10154d60a810001, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,42205,1676174315413 2023-02-12 03:59:18,466 ERROR [Listener at localhost.localdomain/46483-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1254169d rejected from java.util.concurrent.ThreadPoolExecutor@7113da63[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 03:59:18,466 ERROR [Listener at localhost.localdomain/46483-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7bd4c5b1 rejected from java.util.concurrent.ThreadPoolExecutor@3f1eac6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 03:59:18,466 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a61f0c4967b34772a90bac0e6119b40f 2023-02-12 03:59:18,466 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/table/a61f0c4967b34772a90bac0e6119b40f, entries=22, sequenceid=307, filesize=7.0 K 2023-02-12 03:59:18,467 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~137.88 KB/141192, heapSize ~212.70 KB/217808, currentSize=0 B/0 for 1588230740 in 105ms, sequenceid=307, compaction requested=false 2023-02-12 03:59:18,470 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/data/hbase/meta/1588230740/recovered.edits/310.seqid, newMaxSeqId=310, maxSeqId=1 2023-02-12 03:59:18,470 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-12 03:59:18,471 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-12 03:59:18,472 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676174358362Running coprocessor pre-close hooks at 1676174358362Disabling compacts and flushes for region at 1676174358362Disabling writes for close at 1676174358362Obtaining lock to block concurrent updates at 1676174358362Preparing flush snapshotting stores in 1588230740 at 1676174358362Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=141192, getHeapSize=217808, getOffHeapSize=0, getCellsCount=997 at 1676174358363 (+1 ms)Flushing stores of hbase:meta,,1.1588230740 at 1676174358364 (+1 ms)Flushing 1588230740/info: creating writer at 1676174358365 (+1 ms)Flushing 1588230740/info: appending metadata at 1676174358383 (+18 ms)Flushing 1588230740/info: closing flushed file at 1676174358383Flushing 1588230740/rep_barrier: creating writer at 1676174358406 (+23 ms)Flushing 1588230740/rep_barrier: appending metadata at 1676174358409 (+3 ms)Flushing 1588230740/rep_barrier: closing flushed file at 1676174358409Flushing 1588230740/table: creating writer at 1676174358429 (+20 ms)Flushing 1588230740/table: appending metadata at 1676174358432 (+3 ms)Flushing 1588230740/table: closing flushed file at 1676174358432Flushing 1588230740/info: reopening flushed file at 1676174358452 (+20 ms)Flushing 1588230740/rep_barrier: reopening flushed file at 1676174358456 (+4 ms)Flushing 1588230740/table: reopening flushed file at 1676174358460 (+4 ms)Finished flush of dataSize ~137.88 KB/141192, heapSize ~212.70 KB/217808, currentSize=0 B/0 for 1588230740 in 105ms, sequenceid=307, compaction requested=false at 1676174358467 (+7 ms)Writing region close event to WAL at 1676174358468 (+1 ms)Running coprocessor post-close hooks at 1676174358470 (+2 ms)Closed at 1676174358471 (+1 ms) 2023-02-12 03:59:18,472 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-12 03:59:18,562 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,42205,1676174315413] 2023-02-12 03:59:18,562 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,42205,1676174315413; numProcessing=1 2023-02-12 03:59:18,563 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,42205,1676174315413 already deleted, retry=false 2023-02-12 03:59:18,564 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,42205,1676174315413 expired; onlineServers=2 2023-02-12 03:59:18,564 INFO [RS:1;jenkins-hbase20:35783] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,35783,1676174315451; all regions closed. 2023-02-12 03:59:18,564 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,35785,1676174315481] 2023-02-12 03:59:18,564 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,35785,1676174315481; numProcessing=2 2023-02-12 03:59:18,565 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,35785,1676174315481 already deleted, retry=false 2023-02-12 03:59:18,565 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,35785,1676174315481 expired; onlineServers=1 2023-02-12 03:59:18,572 DEBUG [RS:1;jenkins-hbase20:35783] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/oldWALs 2023-02-12 03:59:18,572 INFO [RS:1;jenkins-hbase20:35783] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C35783%2C1676174315451.meta:.meta(num 1676174316975) 2023-02-12 03:59:18,577 DEBUG [RS:1;jenkins-hbase20:35783] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/oldWALs 2023-02-12 03:59:18,577 INFO [RS:1;jenkins-hbase20:35783] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C35783%2C1676174315451:(num 1676174316836) 2023-02-12 03:59:18,578 DEBUG [RS:1;jenkins-hbase20:35783] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:59:18,578 INFO [RS:1;jenkins-hbase20:35783] regionserver.LeaseManager(133): Closed leases 2023-02-12 03:59:18,578 INFO [RS:1;jenkins-hbase20:35783] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-12 03:59:18,578 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-12 03:59:18,579 INFO [RS:1;jenkins-hbase20:35783] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:35783 2023-02-12 03:59:18,582 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:35783-0x10154d60a810002, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,35783,1676174315451 2023-02-12 03:59:18,582 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-12 03:59:18,582 ERROR [Listener at localhost.localdomain/46483-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@17bfc6e0 rejected from java.util.concurrent.ThreadPoolExecutor@138050cf[Shutting down, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 03:59:18,583 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,35783,1676174315451] 2023-02-12 03:59:18,583 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,35783,1676174315451; numProcessing=3 2023-02-12 03:59:18,584 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,35783,1676174315451 already deleted, retry=false 2023-02-12 03:59:18,584 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,35783,1676174315451 expired; onlineServers=0 2023-02-12 03:59:18,584 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,37305,1676174314460' ***** 2023-02-12 03:59:18,584 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-12 03:59:18,585 DEBUG [M:0;jenkins-hbase20:37305] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3dc8cb6f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-02-12 03:59:18,585 INFO [M:0;jenkins-hbase20:37305] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,37305,1676174314460 2023-02-12 03:59:18,585 INFO [M:0;jenkins-hbase20:37305] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,37305,1676174314460; all regions closed. 2023-02-12 03:59:18,585 DEBUG [M:0;jenkins-hbase20:37305] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 03:59:18,585 DEBUG [M:0;jenkins-hbase20:37305] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-12 03:59:18,585 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-12 03:59:18,585 DEBUG [M:0;jenkins-hbase20:37305] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-12 03:59:18,585 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1676174316481] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1676174316481,5,FailOnTimeoutGroup] 2023-02-12 03:59:18,585 INFO [M:0;jenkins-hbase20:37305] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-12 03:59:18,585 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1676174316480] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1676174316480,5,FailOnTimeoutGroup] 2023-02-12 03:59:18,586 INFO [M:0;jenkins-hbase20:37305] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-12 03:59:18,586 INFO [M:0;jenkins-hbase20:37305] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-02-12 03:59:18,586 DEBUG [M:0;jenkins-hbase20:37305] master.HMaster(1502): Stopping service threads 2023-02-12 03:59:18,587 INFO [M:0;jenkins-hbase20:37305] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-12 03:59:18,587 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-12 03:59:18,587 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-12 03:59:18,587 ERROR [M:0;jenkins-hbase20:37305] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] Thread[HFileArchiver-9,5,PEWorkerGroup] Thread[HFileArchiver-10,5,PEWorkerGroup] 2023-02-12 03:59:18,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-12 03:59:18,588 INFO [M:0;jenkins-hbase20:37305] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-12 03:59:18,588 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-12 03:59:18,588 DEBUG [M:0;jenkins-hbase20:37305] zookeeper.ZKUtil(398): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-12 03:59:18,588 WARN [M:0;jenkins-hbase20:37305] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-12 03:59:18,588 INFO [M:0;jenkins-hbase20:37305] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-12 03:59:18,589 INFO [M:0;jenkins-hbase20:37305] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-12 03:59:18,589 DEBUG [M:0;jenkins-hbase20:37305] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-12 03:59:18,589 INFO [M:0;jenkins-hbase20:37305] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-12 03:59:18,589 DEBUG [M:0;jenkins-hbase20:37305] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-12 03:59:18,589 DEBUG [M:0;jenkins-hbase20:37305] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-12 03:59:18,589 DEBUG [M:0;jenkins-hbase20:37305] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-12 03:59:18,589 INFO [M:0;jenkins-hbase20:37305] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=844.65 KB heapSize=1.01 MB 2023-02-12 03:59:18,609 INFO [M:0;jenkins-hbase20:37305] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=844.65 KB at sequenceid=2048 (bloomFilter=true), to=hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/88e7d78a7f734260a2bdc184eb5319f2 2023-02-12 03:59:18,612 INFO [M:0;jenkins-hbase20:37305] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 88e7d78a7f734260a2bdc184eb5319f2 2023-02-12 03:59:18,613 DEBUG [M:0;jenkins-hbase20:37305] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/88e7d78a7f734260a2bdc184eb5319f2 as hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/88e7d78a7f734260a2bdc184eb5319f2 2023-02-12 03:59:18,615 INFO [M:0;jenkins-hbase20:37305] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 88e7d78a7f734260a2bdc184eb5319f2 2023-02-12 03:59:18,615 INFO [M:0;jenkins-hbase20:37305] regionserver.HStore(1080): Added hdfs://localhost.localdomain:46179/user/jenkins/test-data/ebb12079-0c23-1514-b253-8fcc5fca790a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/88e7d78a7f734260a2bdc184eb5319f2, entries=282, sequenceid=2048, filesize=30.7 K 2023-02-12 03:59:18,616 INFO [M:0;jenkins-hbase20:37305] regionserver.HRegion(2947): Finished flush of dataSize ~844.65 KB/864925, heapSize ~1.01 MB/1056704, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 27ms, sequenceid=2048, compaction requested=false 2023-02-12 03:59:18,617 INFO [M:0;jenkins-hbase20:37305] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-12 03:59:18,617 DEBUG [M:0;jenkins-hbase20:37305] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676174358589Disabling compacts and flushes for region at 1676174358589Disabling writes for close at 1676174358589Obtaining lock to block concurrent updates at 1676174358589Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676174358589Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=864925, getHeapSize=1056704, getOffHeapSize=0, getCellsCount=2535 at 1676174358589Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676174358590 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676174358590Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676174358596 (+6 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676174358596Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676174358613 (+17 ms)Finished flush of dataSize ~844.65 KB/864925, heapSize ~1.01 MB/1056704, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 27ms, sequenceid=2048, compaction requested=false at 1676174358616 (+3 ms)Writing region close event to WAL at 1676174358617 (+1 ms)Closed at 1676174358617 2023-02-12 03:59:18,622 INFO [M:0;jenkins-hbase20:37305] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-12 03:59:18,623 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-12 03:59:18,623 INFO [M:0;jenkins-hbase20:37305] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:37305 2023-02-12 03:59:18,624 DEBUG [M:0;jenkins-hbase20:37305] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,37305,1676174314460 already deleted, retry=false 2023-02-12 03:59:18,756 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-12 03:59:18,756 INFO [M:0;jenkins-hbase20:37305] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,37305,1676174314460; zookeeper connection closed. 2023-02-12 03:59:18,757 ERROR [Listener at localhost.localdomain/46483-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4db533a5 rejected from java.util.concurrent.ThreadPoolExecutor@7d9623ce[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 27] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 03:59:18,757 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): master:37305-0x10154d60a810000, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-12 03:59:18,757 ERROR [Listener at localhost.localdomain/46483-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4e9afe41 rejected from java.util.concurrent.ThreadPoolExecutor@7d9623ce[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 27] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 03:59:18,857 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:35783-0x10154d60a810002, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-12 03:59:18,857 INFO [RS:1;jenkins-hbase20:35783] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,35783,1676174315451; zookeeper connection closed. 2023-02-12 03:59:18,857 ERROR [Listener at localhost.localdomain/46483-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@effb077 rejected from java.util.concurrent.ThreadPoolExecutor@138050cf[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 03:59:18,857 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3d4adc2c] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3d4adc2c 2023-02-12 03:59:18,857 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:35783-0x10154d60a810002, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-12 03:59:18,857 ERROR [Listener at localhost.localdomain/46483-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7689dd8a rejected from java.util.concurrent.ThreadPoolExecutor@138050cf[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 03:59:18,957 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:42205-0x10154d60a810001, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-12 03:59:18,957 INFO [RS:0;jenkins-hbase20:42205] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,42205,1676174315413; zookeeper connection closed. 2023-02-12 03:59:18,957 ERROR [Listener at localhost.localdomain/46483-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6a409a65 rejected from java.util.concurrent.ThreadPoolExecutor@7113da63[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 03:59:18,958 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:42205-0x10154d60a810001, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-12 03:59:18,958 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@62b7504a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@62b7504a 2023-02-12 03:59:18,958 ERROR [Listener at localhost.localdomain/46483-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@66eca18e rejected from java.util.concurrent.ThreadPoolExecutor@7113da63[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 03:59:19,057 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:35785-0x10154d60a810003, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-12 03:59:19,057 INFO [RS:2;jenkins-hbase20:35785] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,35785,1676174315481; zookeeper connection closed. 2023-02-12 03:59:19,058 ERROR [Listener at localhost.localdomain/46483-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@373300bc rejected from java.util.concurrent.ThreadPoolExecutor@3f1eac6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 03:59:19,058 DEBUG [Listener at localhost.localdomain/46483-EventThread] zookeeper.ZKWatcher(600): regionserver:35785-0x10154d60a810003, quorum=127.0.0.1:50368, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-12 03:59:19,058 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6d9cb303] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6d9cb303 2023-02-12 03:59:19,058 ERROR [Listener at localhost.localdomain/46483-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@60d04e7c rejected from java.util.concurrent.ThreadPoolExecutor@3f1eac6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 03:59:19,059 INFO [Listener at localhost.localdomain/46483] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-12 03:59:19,060 WARN [Listener at localhost.localdomain/46483] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-12 03:59:19,067 INFO [Listener at localhost.localdomain/46483] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-12 03:59:19,175 WARN [BP-169146336-148.251.75.209-1676174311436 heartbeating to localhost.localdomain/127.0.0.1:46179] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-12 03:59:19,177 WARN [BP-169146336-148.251.75.209-1676174311436 heartbeating to localhost.localdomain/127.0.0.1:46179] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-169146336-148.251.75.209-1676174311436 (Datanode Uuid e4e7b76d-7663-47f8-a739-560eaae89347) service to localhost.localdomain/127.0.0.1:46179 2023-02-12 03:59:19,181 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/cluster_9ba13d3e-ed14-b1d2-0f74-8c456c2a06ab/dfs/data/data5/current/BP-169146336-148.251.75.209-1676174311436] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-12 03:59:19,181 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/cluster_9ba13d3e-ed14-b1d2-0f74-8c456c2a06ab/dfs/data/data6/current/BP-169146336-148.251.75.209-1676174311436] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-12 03:59:19,183 WARN [Listener at localhost.localdomain/46483] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-12 03:59:19,184 INFO [Listener at localhost.localdomain/46483] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-12 03:59:19,291 WARN [BP-169146336-148.251.75.209-1676174311436 heartbeating to localhost.localdomain/127.0.0.1:46179] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-12 03:59:19,291 WARN [BP-169146336-148.251.75.209-1676174311436 heartbeating to localhost.localdomain/127.0.0.1:46179] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-169146336-148.251.75.209-1676174311436 (Datanode Uuid f058bc0e-c917-4e0e-b523-a6533f21d9b8) service to localhost.localdomain/127.0.0.1:46179 2023-02-12 03:59:19,293 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/cluster_9ba13d3e-ed14-b1d2-0f74-8c456c2a06ab/dfs/data/data3/current/BP-169146336-148.251.75.209-1676174311436] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-12 03:59:19,293 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/cluster_9ba13d3e-ed14-b1d2-0f74-8c456c2a06ab/dfs/data/data4/current/BP-169146336-148.251.75.209-1676174311436] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-12 03:59:19,294 WARN [Listener at localhost.localdomain/46483] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-12 03:59:19,296 INFO [Listener at localhost.localdomain/46483] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-12 03:59:19,401 WARN [BP-169146336-148.251.75.209-1676174311436 heartbeating to localhost.localdomain/127.0.0.1:46179] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-12 03:59:19,401 WARN [BP-169146336-148.251.75.209-1676174311436 heartbeating to localhost.localdomain/127.0.0.1:46179] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-169146336-148.251.75.209-1676174311436 (Datanode Uuid d006747b-34dc-492d-8dd2-01dc9639e0d9) service to localhost.localdomain/127.0.0.1:46179 2023-02-12 03:59:19,402 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/cluster_9ba13d3e-ed14-b1d2-0f74-8c456c2a06ab/dfs/data/data1/current/BP-169146336-148.251.75.209-1676174311436] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-12 03:59:19,403 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3c9f01b9-722b-1bf3-c24b-7fc6c70287e5/cluster_9ba13d3e-ed14-b1d2-0f74-8c456c2a06ab/dfs/data/data2/current/BP-169146336-148.251.75.209-1676174311436] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-12 03:59:19,431 INFO [Listener at localhost.localdomain/46483] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-02-12 03:59:19,553 INFO [Listener at localhost.localdomain/46483] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-12 03:59:19,628 INFO [Listener at localhost.localdomain/46483] hbase.HBaseTestingUtility(1293): Minicluster is down